Let's suppose that an object sits in water. The light rays comings from that object will get refracted at the water surface at different angles and the refracted rays are not concurrent (do not meet at the same point). How is it possible for us to see a virtual image if the light rays are not concurrent at any given point (or, said otherwise, if the light rays meets at many different points) ?
2 Answers
Only a narrow beam of light rays can enter to your pupils, the virtual image can be regarded as the intersection of two closely refracted rays.
With some calculus, the locus of the virtual image formed is the envelope of the refracted rays, which is known as diacaustic.
See more on caustic.
The apparent depth formula holds good for normal incidence only:
$$n=\frac{\text{real depth}}{\text{apparent depth}}$$
Picture below shows the image distortion of pool bottom marks running in "horizontal" and "vertical" directions.
An ideal lens forms an image of an object whose location everyone can agree upon*. An object under water is different in the sense that the image location depends on the location of the observer, i.e. the angle at which you look at it. This is caused precisely by your observation that rays that originate from the object appear to emanate from different positions under water for different angles of refraction.
This doesn't mean there is no image to speak of. You clearly do see an image of an object submerged under water. It just appears to move as you look at it from different directions. A collection of rays with roughly the same angle of refraction will still appear to come from the same point, more or less (note that the rays that end up entering your eye likewise differ very little in direction). But the location of this point depends on the angle of refraction. This effect is also responsible for large objects under water appearing "distorted", since the rays from different points on the object reach you from different directions.
*For typical lenses, the image location will still appear to change somewhat for objects or observers far from the optical axis, as a result of e.g. spherical aberration.