# How to extract depth of a pixel from an image?

I'll have a few pictures that show the same scene, but only a few things move around a little bit. Now I don't have an information about any size in real life but I'd like to extract a scaling factor from these scenario.

I know when a 3D scene (with X,Y,Z coordinates) is captured in a 2D picture (with X/Z,Y/Z coordinates), it ain't easy to calculate the depth of each object/pixel but becomes easier if one size is given (e.g. diameter of a ball).

But is there still a way to determine the real size of the objects in the scene, without knowing a scaling factor ?

[Update]:

Imagine a Ball is thrown into the scene. Since I know gravitation and the time difference between each pixel, could this be a way too? I mean the horizontal and vertical trajectory are independend from each other, that's why I could retrieve the way in direction of the gravitation axis. But the probleme I see here, is that when the ball hits the ground is then more upwards in the picture, the more he gets in the back - I believe this makes it hard to distingush the axis from each other again, and so to tell, which direction is now "to the ground". Or am I going wrong here?

• Scene reconstruction is a hard task. Search for "volumetric scene reconstruction" and you will find research on this topic, although it belongs more into computer science/computer graphics than into physics. Jun 25, 2015 at 3:15

Generally and fundamentally, the answer to your question is no. Photography / 2D imaging is the prototypical example of the mathematical notion of projection, which, by its definition, destroys information by extracting "class" information from sets of objects: forgetting about fine differences and reporting only the class. Here, for example, if $z$ is the direction away from the plane of the photograph, a photograph can be roughly described as registering of the $(x, y)$ co-ordinates of each of the objects in the photo, but forgetting about the $z$. Two bright objects at a point $(x,y)$ will yield the same pixel, regardless of their $z$ co-ordinate.

Sometimes you can use further information to extract the $z$. Astronomers must do this all the time through the use of standard candles to infer a the distance to an object of known brightness from its apparent brightness, use of redshift and so forth. See the Cosmic Distance Ladder Wikipedia Page.

Othertimes the photographic method takes specific steps to encode the $z$ information in the 2D representation, and generally one must make an assumption about the kind of object being photographed: commonly that $z$ is a continuous function of $(x,\,y)$, such as when you're looking down from an aircraft onto the land's surface. For example, Structured 3D Light Scanning casts bright, straight fringes onto the object of known spacing and angle, the object's deviation from flatness warps the otherwise straight fringe and the $z$ co-ordinate can be calculated from this essentially triangulation data. Photogrammetry is somewhat similar in principle but very different in technique: it uses stereo photography and makes distance inference by triangulation with the different $(x,y)$ co-ordinates of the same object in two different photographs taken from different points.

This is true, but telling whether something is in focus can be harder than you might think. There needs to be fine, known structure in the image for a definitive test, especially if you want the method to be an automated image processing procedure. And the deduction of the z information for everything in the image can only be done with a 3D stack of photos, which are then run through a point spread function deconvolution algorithm (this is the idea of "deconvolution microscopy". The use of coherent light, interferometry and phase reconstruction algorithms is another method to map surfaces with high precision, but this is limited to objects whose $z$ co-ordinate varies by fewer than about 100 wavelengths or so. The phase camera is a method in between straight interferometry and 3D stack deconvolution.