The easiest way to see it is to replace the rod with two point masses connected by a rigid bar. This will simulate two opposite small bits of the rod. Let all three masses have mass $m$, the distance from the single mass to the center of the bar $R$ and the distance from the center of the bar to one of the masses be $r$. Assume the bar is oriented along the line from the center to the single mass. Then $$F=Gm^2\left(\frac 1{(R-r)^2}+\frac 1{(R+r)^2}\right)$$ while lumping the two masses at the center of the bar gives $$F=Gm^2\left(\frac 2{R^2}\right)$$ and we have $$\left(\frac 1{(R-r)^2}+\frac 1{(R+r)^2}\right)=\frac{2R^2+2r^2}{(R^2-r^2)^2}\gt \frac 2{R^2}$$ The point is that the force on the close mass increases more than the force on the far mass decreases. If you do the calculation with the bar perpendicular to the line to the single mass, you will find the force is reduced because the distance to the two masses is $\sqrt{R^2+r^2}$ instead of $R$
If $r \ll R$ these effects become small and may be able to be ignored in practical calculation. The corrections fall off as $\frac 1{R^3}$ or faster. From far enough away, you can just lump all the mass at the cm and consider a body a point mass.