I would think, in fact, that the relative sizes don't particularly
matter. Why doesn't gravity make things accelerate past the speed of
light if they are far enough away?
To be sure (just in case), please keep in mind that the (Newtonian) gravitational force is not constant with distance even though we usually approximate it as constant for elementary problems involving relatively small objects falling near the surface of Earth.
Consider a gravitating, spherical body with some non-zero radius, and consider a much smaller object that falls towards that body from rest a great distance away. We can calculate the speed with which the object impacts the surface. This speed is equal to the escape velocity at the surface of the body.
So, your question can be recast to something like this:
Why are the no bodies with escape velocity $v_e$ greater than the speed of light?
Note that such a body with $v_e \gt c$ would necessarily trap light. In the Newtonian context, where the speed $c$ isn't a speed limit, an object falling from a great distance would impact the surface of the body with speed greater than the speed of light. See, for example, Can a black hole be explained by newtonian gravity?
But in the relativistic context, no massive object can have relative speed $v \ge c$. So, if a massive body contracts to the radius such that the escape velocity at the surface is $c$, the body must continue to collapse leaving an event horizon from which the 'escape' velocity is precisely $c$
However, now we're in a highly curved spacetime where it's often very difficult if not impossible to properly define concepts that were straightforward in the Newtonian context.
For example, you might from the above conclude that an object falling from a great distance towards a black hole would have speed $c$ at the event horizon. However, in a curved spacetime, where the clocks and rods of different observers outside of and at rest relative to the black hole are not the same, it isn't at all clear what such a statement would mean. In fact, it turns out that no object actually reaches the event horizon according to these observers!