# Extracting the 3D coordinates of a moving object from a video

Take a look at these two pictures, which are stills from a video which demonstrates magnus effect in football:  I want to extract the coordinates of this ball in 3D space from this video. These are the steps I intend to use:

1. The ball is initially 1 m away from the camera. I can use this information to calculate the distance from camera in the later frames. (with it's angular diameter)

2. A football is 22 cm across. This can be used to calculate a quantity which I'm calling anglePerPixel(which is 22/100/<initial width in pixels>. It can be used to calculate the angle of elevation of the ball from the horizon.

3. Imagine a plane perpendicular to the ground and the camera direction, which cuts the camera view in two equal parts. It will appear as a line in the camera view. We can measure perpendicular distance of the ball from this plane in ball units, by measuring how many footballs we can fit between this plane and our football.

These 3 independent coordinates could be used to calculate and plot the path of this ball, i.e., if this procedure was correct, which it isn't.

I'm confident that the first step is correct. The second step yields incorrect results(about half of the expected value). The third step also looks correct to me.

How do I fix the second step? (and any mistake in the other two steps, if there's any)

Edit:

It's possible to use the method of second step to calculate the elevation as well, but it won't be very accurate since the camera is about 30 cm above ground and is aimed about 3-4 degrees above the horizon.

Maybe we could calculate the position of ball relative to the direction of the camera (instead of the ground) and try to translate it once it's done.

If you are looking for geometric accuracy, your approach is quite vague and possibly not completely accurate. I also have some doubts about your initial assumptions.

Let's say indeed, the ball's diameter is $$22$$ cm and the distance, along the straight line on the ground, connecting the camera's position to the place where the ball touches the ground, is $$100$$ cm. From what I am seeing on the first photo, the camera is not $$30$$ cm above ground. Otherwise it would be looking at the ball (which is $$22$$ cm in diameter) from above (because $$22 < 30$$), while in fact, we see that the camera is looking at it a bit from below. More precisely, from what I am seeing, the camera's lower edge is placed almost on the ground and the camera is slightly tilted upwards. So I am going to assume that.

Geometrically the situation is more complex than you assume, because the ball, being originally a sphere, projects onto the photo as an ellipse. I seem to be measuring the height of the ball in the first photo at something like $$3.7$$ cm or so, while the horizontal width seems to be something like $$3.65$$ cm or so. However, I am going to make a simplification, otherwise things are much more complex. I am going to assume that when we are making our calculations, the 3D ball can be represented by a disk of diameter $$22$$ cm always facing the camera fully, i.e. the plane of the disk representing the ball is parallel to the screen $$s$$. I repeat, this is not completely true, this is more of an estimate. Consequently, the results you are going to get are reasonable (I hope :) ) estimates. Intuitively, it seems to me that this approximation leads to a very small error.

First of all, for the most standard camera models, the geometric representation of a camera is a pair, consisting of a point $$O$$ (point of observation) and a screen (a plane $$s$$) not passing through $$O$$. Let $$O_s$$ be the orthogonal projection of $$O$$ onto $$s$$.

Basically, the representation of the three dimensional world onto the two dimensional screen $$s$$ is obtained by connecting the observation point $$O$$ to any other point $$P$$ from the three dimensional space. The intersection point $$P_s$$ of the straight line $$OP$$ with the screen $$s$$ is the two dimensional projection (2D image) of $$P$$ onto the the screen $$s$$.

To be able to calculate relationships between 3D objects and their 2D images, you would need several important parameters of the camera representation. First, you would need to know the location of the point $$O_s$$, the orthogonal projection of $$O$$ onto $$s$$. And second, you would definitely need the distance $$d = |OO_s|$$ between $$O$$ and the plane $$s$$. So to be able to carry out calculations like the ones you want, you need to be able to find the location of point $$O_s$$ and to find somehow $$d$$.

Part 1. By knowing the initial position of the ball and teh camera on photo 1 and by measuring some lengths on the photo, you can deduce the location of $$O_s$$ and the distance $$d=|OO_s|$$.

Part 2. By measuring the coordinates of the center of the ball's image on photo 2 and by measuring/estimating the radius of the ball's image, calculate the 3D position of the ball on the second photo.

Part 1, Step 1. Fix the necessary coordinate systems on the photo and in three dimensions. According to my measurements, the frames of the first and the second photo are the same: identical rectangles with horizontal width $$= 7.1$$ cm and vertical height $$= 14.3$$ cm. This also allows me to assume that the photos have not been cropped. Consequently, since I believe that most cameras' screens are probably designed so that the projection $$O_s$$ of $$O$$ onto the screen coincides with the geometric center of the screen's rectangle $$s$$, point $$O_s$$ should be the intersection point of the diagonals of $$s$$.

Let us introduce the 2D coordinate system $$Lwh$$ to be with origin the lower left corner $$L$$ of the photos, horizontal axis $$L\vec{w}$$ (width) along the lower horizontal edge of the photos, and vertical axis $$O\vec{h}$$ (height) along the left vertical edge of the photos. Next, let us introduce the coordinate system $$O_suv$$ to be with origin point $$O_s$$, horizontal axis $$O_s\vec{u}$$ parallel to the horizontal edge of the photos (and thus to $$L\vec{w}$$), and vertical axis $$O_s\vec{v}$$ parallel to the vertical edge of the photos (and thus to $$L\vec{h}$$). Finally, define the 3D ortho-normal coordinate system $$Oxyz$$, with origin $$O$$, axis $$O\vec{x}$$ parallel to $$O_s\vec{u}$$, axis $$O\vec{z}$$ parallel to $$O_s\vec{v}$$ and axis $$O\vec{y}$$ perpendicular to the screen $$s$$.

Observe axis $$O\vec{z}$$ is not perpendicular to the ground and axis $$O\vec{y}$$ is not parallel to the ground! However, axis $$O\vec{x}$$ is parallel to the ground.

Part 1, Step 2. Find the location of $$O_s$$ in coordinate system $$Lwh$$. Now, in the coordinate system $$Lwh$$ the point $$O_s$$ has coordinates $$(7.1/2, 14.3/2) = (3.55, 7.15)$$. Thus \begin{align} &u = w - 3.55\\ &v = h - 7.15 \end{align}

Part 1, Step 3. Carry out measurements and preliminary constructions on photo 1. Let $$A$$ be the midpoint of the lower edge of the photos and $$B$$ be the midpoint of the upper edge of the photos. Then line $$AB \, || \, L\vec{h} \, || \, O_s\vec{w}$$ and thus $$O_s$$ is the midpoint of $$AB$$ (as well as $$O_s$$ is the intersection point of the diagonals of the photos). My measurements show (more or less) that the image of the ball on photo 1 is symmetric with respect to $$AB$$. Denote by $$D$$ the point where the actual ball touches the ground. By my simplifying assumption I spoke about earlier, the 3D ball is interpreted as a flat disk of diameter $$22$$ cm whose plane is parallel to $$s$$. Then, denote by $$U$$ this disk's diametrically opposite point of $$D$$. Thus $$|DU| = 22$$ cm and $$|AD| = 100$$ cm. See the figure I have added below. Then, if $$D_s = s \cap OD$$ and $$U_s = s \cap OU$$, then $$D_s$$ and $$U_S$$ are respectively the lowest and the highest intersection points of the ball's image with the vertical line $$AB$$ on the screen $$s$$. Thus, we are in the situation of the figure above.

Point $$D'$$ on $$DU$$ is such that $$O_sD'$$ is parallel to $$AD$$, i.e. $$O_sD' \, || \, AD$$, and since by assumption $$AB \, || \, DU$$, the quad $$ADD'O_s$$ is a parallelogram so $$|DD'| = |AO_s|$$. Point $$D$$ is $$C = OO_s \cap DU$$. Therefore, triangle $$O_sD'C$$ is right angled with $$\angle \, O_sCD' = 90^{\circ}$$ because $$OO_s$$ is orthogonal to $$AB$$ and $$AB$$ is parallel to $$DU$$.

I measured on photo 1 that \begin{align} &|AD_s| = 4.3 \text{ cm }\\ &|AO_s| = \frac{1}{2}\, |AB| = 7.15 \text{ cm }\\ &|AU_s| = 8 \text{ cm }\\ \end{align} Consequently, \begin{align} &|D_sO_s| = 7.15 - 4.3 = 2.85 \text{ cm }\\ &|O_sU_s| = 8 - 7.15 = 0.85 \text{ cm }\\ &|D_sU_s| = 8 - 4.3 = 3.7 \text{ cm }\\ \end{align}

Part 1, Step 4. Calculate $$d=|OO_s|$$. By Thales' intercept theorem (or similarity of triangles if you prefer) $$\frac{|DC|}{|DU|} = \frac{|D_sO_s|}{|D_sU_s|}$$ $$\frac{|DC|}{22} = \frac{2.85}{3.7}$$ $$|DC| = \frac{2.85 \cdot 22}{3.7} = 16.95 \text{ cm }$$ Thus $$|D'C| = |DC| - |DD'| = |DC| - |AO_s| = 16.95 - 7.15 = 9.8 \text{ cm }$$ By Pythagoras' theorem for right triangle $$O_sD'C$$ we find $$|O_sC| = \sqrt{|O_sD'|^2 - |D'C|^2} = \sqrt{|AD|^2 - |D'C|^2} = \sqrt{100^2 - 9.8^2} = 99.52 \text{ cm }$$ and we can even calculate the angle $$\theta = \angle D'O_sC = \arcsin{\frac{|D'C|}{|O_sD'|}} = \arcsin{\frac{9.8}{100}} = 5.624^{\circ}$$ which shows how much the camera is tilted relative to the ground. Finally, again by Thales' theorem or similarity $$\frac{|OO_s|}{|OC|} = \frac{|OO_s|}{|OO_s| + |O_sC|} = \frac{d}{d+99.52} = \frac{|D_sO_s|}{|DC|} = \frac{2.85}{16.95}$$ which when we solve for $$d$$, gives $$d = |OO_s| = 20.12 \text{ cm}$$

Part 2, Step 1. Measuring the location of the center and the radius of the ball's image on photo 2. Since the image of the ball on photo 2 is almost a circular disc (we assume that because of the earlier assumption that the real ball is represented by a disk parallel to $$s$$), I measured the distance between the lower edge of photo 2 and the lowest point from the ball's image and found that it is $$h_l = 10$$ cm. Similarly, the distance between the lower edge of photo 2 and the uppermost point from the ball's image is $$h_u = 10.3$$ cm. The horizontal distance between the left vertical edge of photo 2 (axis $$L\vec{h}$$) and either of the two points, mentioned in the previous sentence, is $$w_2 = 3.87$$ cm. Thus, the coordinates of the center $$Q_2$$ of the ball's image on photo 2 with respect to the coordinate system $$Lwh$$ are approximately $$Q_2 = \big(w_2, \, (h_u+h_l)/2\big) = \big(3.87, \, (10.3+10)/2\big) = \big( 3.87, \, 10.15\big)$$ and the diameter of the ball's image on photo 2 is $$h_u-h_l = 0.3$$ cm.

For future reference, let us denote by $$Q$$ the 3D center of the real ball in the case of photo 2.

Part 2, Step 2. Calculating the 3D coordinates of $$Q$$ with respect to the coordinate system $$Oxyz$$. To that end, we work only with photo 2. By the simplifying assumption from before, we assume that the image of the real ball on the screen $$s$$ is a circular disk (we call it image disk), and at the same time the real ball in 3D is represented by a circular disk, parallel to $$s$$ (we call it real disk). Therefore the center of the real disk, which is $$Q$$, the center of the image disk $$Q_2$$ and the point $$O$$ are collinear. Moreover, the image disk and the real disk are (by assumption) homothetic to each other from the origin $$O$$. In other words, there is a similarity transformation (a stretching of 3D space with respect to point $$O$$) which maps the image disk to the real disk, so in particular it maps the center $$Q_2$$ to the center $$Q$$, while keeping the origin $$O$$ of $$Oxyz$$ fixed. Thus, the coefficient of similarity (the coefficient of stretching) is $$\lambda = \frac{\text{ diameter of real disk }}{\text{ diameter of image disk }} = \frac{ 22 }{ 0.3 } = 220/3 = 73.33$$ The coordinates of $$Q_2$$ with respect to coordinate system $$O_suv$$ are simply \begin{align} &u_2 = 3.87 - 3.55 = 0.32 \text{ cm}\\ &v_2 = 10.15 - 7.15 = 3 \text{ cm} \end{align} Consequently, the 3D coordinates of point $$Q_2$$ with respect to $$Oxyz$$ are $$Q_2 = \big(u_2,\, d,\, v_2\big) = \big(0.32,\, 20.12 ,\, 3 \big)$$ Therefore, to obtain the coordinates of $$Q$$ we simply have to multiply the coordinates of $$Q_2$$ by the factor $$\lambda$$ and obtain $$Q = \big(\lambda u_2,\, \lambda d ,\, \lambda v_2\big) = \big(0.32 \cdot 220/3 ,\, 20.12\cdot 220/3 ,\, 3\cdot 220/3\big)$$ Thus, we finally have the coordinates of the center of the real 3D ball from picture 2 $$Q = \big( 23.47,\, 1475.47,\, 220\big)$$ with respect to the coordinate system $$Oxyz$$.

If however, we would like to find the coordinates of $$Q$$ with respect to a coordinate system $$Ox\tilde{y}\tilde{z}$$, where the latter is the rotation of $$Oxyz$$ around axis $$O\vec{x}$$ at an angle of $$- \theta = - \, \angle\, D'O_sC = -\, 5.624^{\circ}$$ so that now not only the axis $$O\vec{x}$$ is parallel to the ground but also the axis $$O\vec{\tilde{y}}$$ is parallel to the ground, while the axis $$O\vec{\tilde{z}}$$ is vertical (orthogonal) to the ground. In order to do that, we simply have to multiply the $$Oxyz-$$coordinates of $$Q$$ by the rotation matrix $$\text{ROT}(\theta) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & \cos{\theta} & -\,\sin{\theta} \\ 0 & \sin{\theta} & \cos{\theta} \\ \end{pmatrix}= \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0.995 & -\,0.098 \\ 0 & 0.098 & 0.995 \\ \end{pmatrix}$$ and obtain the $$Ox\tilde{y}\tilde{z}-$$coordinates of $$Q$$ $$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 0.995 & -\,0.098 \\ 0 & 0.098 & 0.995 \\ \end{pmatrix} \begin{pmatrix} 23.47 \\ 1475.47 \\ 220 \end{pmatrix} = \begin{pmatrix} 23.47\\ 1446.53 \\ 363.5 \end{pmatrix}$$ Furthermore, if we want to calculate how high point $$O$$ is from the ground, we can go back to the geometric figure and see that the height of $$O$$ can be calculated as $$|AO_s| \cos{\theta} - d \sin{\theta} = 7.15 \cdot 0.995 - 20.12 \cdot 0.098 = 5.142 \text{ cm}$$

Finally we can conclude that in the situation depicted on photo 2, the ball has moved from its initial position on photo 1, by approximately $$23.47$$ centimeters to the right, it's height from the ground is approximately $$363.5+5.142 = 368.642$$ centimeters, which is, give or take, $$3$$ meters and $$68$$ centimeters. Horizontally the ball has moved $$14$$ meters and $$47$$ centimeters from point $$O$$, which is roughly $$13$$ meters from it's initial position.

• Great, where can I learn more about the camera model you used in this solution? – saga Nov 9 '18 at 6:01
• @saga All I used in my answer was exactly the classical perspective representation of a screen and a point (which I have described in the answer). After that it was all Euclidean geometry. However, if you google perspective projection, pinhole camera, computer graphics you should find notes, articles and sites. See for example the video youtube.com/watch?v=17kqhGRDHc8 or there is this idav.ucdavis.edu/education/GraphicsNotes/Camera-Transform/… – Futurologist Nov 9 '18 at 16:24
• In Part1 Step4, you use this equation: |D'C| = |DC| - |DD'| = |DC| - |AD_s| = 16.95 - 4.3 = 12.65. You got 4.3cm by measurement from the picture, isn't it possible that the image might be scaled? – saga Nov 17 '18 at 11:20
• @saga Of course the images can be scaled: both of them with the same scale factor. Bit with the limited information provided, I can work with the best that I have. You do not happen to know the parameters of the camera, do you? The best thing to know are the parameters of the camera: the distance $d$ and the resolution (pixels and how big a pixel is in mm or something like that). However, for a lot of the computations, I have used similarity and fractions of lengths, which absorb scaling factors. There could be some mild variations in the depth coordinate ($y-$coordinate). – Futurologist Nov 18 '18 at 1:14
• @saga However, the methodology and the geometry, not the actual numbers, are the important message. I strongly suggest to replace all the lengths measured on both photos with the same numbers multiplied by an unknown factor $\mu$. Then you will see which parameters will vary with respect to scaling $\mu$ and which do not. For example, $d=d(\mu)$ and the angle $\theta = \theta(\mu)$. Also, you can see how much $\mu$ really impacts the parameters. – Futurologist Nov 18 '18 at 1:21

There are a couple of possible errors associated with step 2. For instance, the formula for the angle of elevation actually gives the tangent of the angle, not the angle per se, so it would be accurate for small angles only. But this error should be insignificant and could be easily corrected by changing the formula.

Another possible source of error in step 2 could be due to the elevation of the camera.

Based on the first picture, we can infer that the camera is mounted above the equatorial plane of the soccer ball. This is because the line of horizon (the edge of the field) is above the center of the ball. If the camera was mounted at the same level as the center of the ball, the line of horizon would be below the center of the ball. This is illustrated on the diagram below with a blue line. As a result of such placement of the camera, the estimated elevation angle could be greater than the actual elevation angle. The angles, $$\theta '$$ and $$\theta$$, are shown on the diagram below. Even if the camera was properly aligned with the ball, the estimated angle would not be quite accurate, due to the $$1$$ meter offset between the ball and the camera, but the error here would not be significant.

I think that for estimating the elevation of the ball, you could use the same method you are using for estimating the lateral displacement of the ball in step 3. You could use ground as a reference plane: the lower the position of the camera, the smaller the error will be.

• I'm using the initial position of the ball only for the reference. First and last second of the flight are being ignored, so the misalignment of camera wouldn't be a problem. I can't see how the method of step 3 could be used to calculate elevation, can you elaborate on that a bit? – saga Nov 3 '18 at 18:03
• Wouldn't we need the position of projection of ball on ground for accurately calculating the elevation with that method, am I misunderstanding something? – saga Nov 3 '18 at 18:05
• @saga If the camera is mounted close to ground, ideally, at ground level, the ground plane will divide the view, vertically, in two halves, similar to the vertical plane, described in step 3, divideing the view in two halves horizontally. From that position, the view of the ground will be significantly comressed, ideally, to a line, so the error will be small. – V.F. Nov 3 '18 at 18:48

For step 1, which you did not describe, you need to include a factor that takes into account that you are not measuring actual angles but a size in a picture. The geometry is like the picture below (sorry for the terrible drawing). if we call $$R$$=distance to the ball, $$R'$$=distance to the picture, $$S$$=size of the ball, $$S'$$ size of the ball in the picture, then you have:

$$\theta=S/r=S'/r'=>S/(R/ \cos \alpha)=S'/(R'/ \cos \alpha)$$

so that

$$R=SR'/S'$$

with the initial distance you can calculate $$R'$$, and then $$R$$ for any distance, using the formula $$R=SR'/S'$$. S is the arc, but is a reasonable approximation to the ball's projection, but you can make it more precise if you want

For step 2 you can calculate the height but you need either the height of the camera, $$y_0$$, or the distance from the camera to the bottom of the picture, R_0. If $$y$$ is the actual height of the ball from the floor, $$y_0$$ the height of the camera, $$R$$ and $$R'$$ the same as before, $$y'$$ and $$y'_0$$ are the height of the ball in the picture (in pixels) and the height in the picture of an object at the height of the camera (which is independent of the distance), then you have:

$$\frac{y-y_0}{R}=\frac{y'-y'_0}{R'}$$ (1)

with

$$y'_0=y_0R'/R_0$$ (2)

Step 3 is the same as step (2) only that here $$y'_0=0$$  • I'm having trouble understanding the equations. y is the actual height,, height of what? This is how I'm looking at the equations: LHS of equation 1 is the sine of the angle of elevation, how does that equal the rhs? Maybe another pic would help me understand the solution ;-) – saga Nov 6 '18 at 5:46
• I've added some more details to the post. – saga Nov 6 '18 at 5:46
• I just edited it, I hope is more clear now – Wolphram jonny Nov 6 '18 at 19:56