I have wondered the exact same thing myself when going through this chapter and other optics textbook (Pedrotti^3)
Strictly speaking, the answer is no, as long as the angle is small the imaging of the equations are quite good.
Broadly speaking however, the off axis distance(in the book usually denoted by variable $y$) affects just how far you can push the $\theta=\sin\theta$ Approximation. For a very tall object the area around the optical axis will be imaged by a lens appropriately , but beyond some radius, this image will be blurry. (If you press your face near a wall, the periphery becomes blurry for example).
So yes, your understanding that there is an inherent limitation in how off axis you are is correct. However I am always surprised at just how off axis you can be while still holding the small angle approximation. The approximations holds to well within 1% for about 0.15 radians (conservative) so for a 3 meter distance, that is approximately a 45 cm object in radius or 0.9 m in total height. I would consider that quite the tall object at a not particularly far distance. I have done sketch to scale to the best of my ability to illustrate just how large this range can be.
In red I have some 0.9 potato creature
In blue I have a pretty big lens (2m)?
The distance is 3m.
Despite the cartoonish size of the red potato vs the distance, it still qualifies under the paraxial approximation.
In addition, I’d like to note that it’s also possible to have a fairly wide beam even with small angles given large enough distance so you may want to change the statement to something more like “narrow beam for parallel rays”.