Determining the coordinates of the vertices of a triangle from two images taken by two cameras with known position/orientation

Question

Suppose you have a triangle with unknown side lengths hanging in space with unknown position and orientation (i.e. nothing about the dimensions or position or orientation of the triangle is known).

Now you take two images of this triangle by two cameras (or one camera but re-positioned/re-oriented). The camera position/orientation for each image is known, as well as its focal length.

The correspondence between the vertices in the images is known.

From the two images of the triangle, is it possible to compute the coordinates of the vertices of the triangle?

Is there more than one possible set of coordinates?

My approach: is outlined in my answer below.

Answer 1

Let $A_1, A_2, A_3$ be the three vertices in the first image and $B_1, B_2, B_3$ be the corresponding vertices in the second image (these are two-dimensional vectors), and let $V_1, V_2, V_3$ be the corresponding three-dimensional vertices that we're after.

Further, let E_1 be the position of the camera for the first image, and E_2 be the position of the camera for the second image, and let R_1 be the rotation matrix specifying the orientation (with respect to the world) of the camera in the first image and $R_2$ be the rotation matrix specifying the orientation of the camera in the second image.

Then we can write

$ V_1 = E_1 + t_1 R_1 P_1 = E_2 + t_1' R_2 P_1' $

$ V_2 = E_1 + t_2 R_1 P_2 = E_2 + t_2' R_2 P_2' $

$ V_3 = E_1 + t_3 R_1 P_3 = E_2 + t_3' R_2 P_3' $

where

$ P_i = \begin{bmatrix} A_i \\ -z_0 \end{bmatrix} $

and

$ P_i' = \begin{bmatrix} B_i \\ - z_0 \end{bmatrix} $

The above equations are equations of straight lines in space. If the coordinates are exact we can precisely find their intersection and determine $t_1, t_2 , t_3 $ (as well as $t_1' , t_2', t_3'$). If not, then we can use least-squares to find the best value that fits the intersection of the two line pairs for the three vertices.

And that's all we need. Once we have the exact or approximate $t_1, t_2, t_3$ then we have determined $V_1, V_2, V_3$ the world coordinates of the three vertices.

Answer 2

I'm not exactly sure how much information you take as given so two different interpretations.

First, if we know the exact orientation of the image from each picture, then a single photo will give us for each corner of the triangle a half line in space as possible positions. We know the exact direction away from the camera but we don't know how far away the point is. In this case determining the triangle is easy, two pictures from different location provide two different half-lines for each corner of the triangle. They intersect in one point, which is the location of the corner.

Second, if we only know the coordinates of the camera and the angles between the 3 half-lines in one picture but not the exact directions this becomes much more difficult. Two pictures will not be enough to determine the location, my intuitive guess would be 4 pictures are needed but I don't know how to set this up as a system of equations one could solve in a reasonable way.