Start by considering the ordinary Newtonian gravity. This tells us that the acceleration of a test mass due to our planet of mass $M$ is:
$$ a = \frac{GM}{r^2} $$
The acceleration is the rate of change of velocity with time. A fast moving object spends less time near the planet than a slow moving object so its velocity changes less. That means fast moving objects are deflected less than slow moving ones.
Since general relativity reduces to Newtonian gravity when the gravitational fields are small (i.e. everywhere that isn't near a black hole) this also explains why fast moving objects deflect less than slow moving ones in GR.
Showing this rigorously does involve some differential geometry, but I think it's possible to grasp the principle without getting too deeply embedded into the maths. The trajectory followed by the freely falling test mass is described by the geodesic equation:
$$ \frac{\mathrm d^2x^\alpha}{\mathrm d\tau^2} = -\Gamma^\alpha_{\,\,\mu\nu}U^\mu U^\nu \tag{1} $$
This isn't as complicated as it looks (well, not quite!). The left hand side is sort of an acceleration, and the symbols $\Gamma^\alpha_{\,\,\mu\nu}$ are the Christoffel symbols that describe how curved spacetime is. In flat spacetime using the usual $(t,x,y,z)$ coordinates the Christoffel symbols are all zero and our equation becomes:
$$ \frac{\mathrm d^2x^\alpha}{\mathrm d\tau^2} = 0 $$
which is just telling us that in flat spacetime the acceleration is zero i.e. the object travels in a straight line.
In curved spacetime the Christoffel symbols are not zero so we get a non-zero acceleration and the trajectory will be curved, but we still need to explain why the trajectory is different for different velocities. That is simply due to the term $U^\mu$, which is the four-velocity.
So the geodesic equation tells (a) that the path isn't a straight line in curved spacetime and (b) that the amount the path curves by is dependent on the four-velocity $\mathbf U$ of the test mass. That's why test masses moving at different velocities follow different paths.