That the moment of inertia about an axis passing through the CM is minimized, with respect to any other parallel axes, is a consequence of the quadratic (squared) dependence of the moment of inertia on distance. In other words, the ${r^2}$ term in ${I=mr^2}$ makes it so that masses at farther distances are preferentially weighted in their contribution to the overall moment. As you said, for a given object, the moment of inertia will will thus depend on the distribution (distances) of masses about the chosen axis.
For a given torque, one can impart a greater angular acceleration on objects of lesser moment. This is seen in the relation ${\tau=I\alpha}$ (analogous to $F=ma$), where $\alpha$ is angular acceleration. Rearranging, we get $\alpha=\tau/I$, so $\alpha$ is largest when $I$ is smallest.
Intuitively, one can understand how angular acceleration is maximized about the CM by picturing twisting a metal rod. Imagine holding the rod at its end and twisting-- it's difficult. Imagine holding the same rod at its center and twisting-- it's slightly easier.
To describe the above scenario mathematically, we can consider a one dimensional rod of mass $m$ running from ${x=0}$ to $x=l$. The moment of inertia about an axis that runs perpendicular to the rod at $x=0$ (twisting the rod about its end) is given by
$I_{end}=\int_{0}^{l}\rho x^2 dx = \frac{m}{3}l^2$, where $\rho=m/l$ is the mass density of the rod.
The moment about an axis through the middle of the rod, $x=l/2$, is
$I_{mid}=\int_{-l/2}^{l/2}\rho x^2 dx = \frac{m}{12}l^2$.
Note that $I_{mid}<I_{end}$.
Taking a more general approach we could calculate the moment about any perpendicular axis, placed at any value x, as
$I_{x}=\int_{0-x}^{l-x}\rho x^2 dx = \frac{1}{3}\rho(l-x)^3-\frac{1}{3}\rho(0-x)^3$.
This function $I_{x}(x)$ has a minimum at the center of mass when $x=l/2$.