Pinhole camera, rectangle projected to one parallel to axes in image.

Question

I have a situation where I want to take a rectangle in a pin hole camera view and find which the possible corresponding rectangles in 3D space are.

Own work:

My intuition tells me that the set of possible rectangles in the 3D space shall be so that the corner points are parametrized like so:

$(x_k,y_k,1) t_k$

Furthermore, we know that pairwisely $x_1=x_2, x_3=x_4, y_1=y_3, y_2=y_4$, in other words we can reduce down to two points $(x_1,y_1)$ and $(x_4,y_4)$, furthermore we can assume that these are all known.

The plane equation $ax+by+cz + d = 0$ gives 4 parameters a,b,c,d which the parametrized points above need to satisfy.

Now, we can express $t_k\cdot (ax_k+by_k+c) + d = 0$

The problem here is... that this is linear in $t_k$ and in $a,b,c,d$ but separately. Also we have $8$ unknowns and $4$ equations.

Now... for the points to just lie on the same plane is not a condition strong enough. It does not guarantee that the four points will be corners to a rectangle there.

Scalar product of pairwise vectors in 3D space shall be 0. We can express conditions such as : $[x_2t_2-x_1t_1,y_2t_2-y_1t_1,t_2-t_1] \cdot [x_3t_3-x_1t_1,y_3t_3-y_1t_1,t_3-t_1] = 0$

$t_1,t_2,t_3$ will be related by this.

If I am not mistaken it will be a quadratic form which we can seek a null space for. This null space when found will tie a linear scaling of $t_1,t_2,t_3$ to a common parameter, or possible several if the space is bigger than 1 dimension.

But perhaps this procedure will be unnecessarily complicated?

I would like to show something of the sort that a necessary selection of $t_1,t_2,t_3,t_4$ needs to be such that the vectors mentioned at the earliest are all of equal norm. But I am not sure if that is true and if so how to show it.

Answer 1

Since you are using a pinhole camera, if you view an object that is a perfect rectangle in a plane and the image plane is not parallel to that rectangle, the image will have a “vanishing point” for at least two sides of the rectangle, that is, those sides will not be parallel in the image.

Since you have indicated that both the object you view and its image must be rectangles, the object must be parallel to the image plane.

This implies that all four of your $t$ values must be equal.

Simply select a plane parallel to the image plane and project the image to that plane to find a possible set of coordinates for the object.

Answer 2

Let the eye of the observer be at location $ e = (e_x , e_y, 0)$. And let the four points of the quadrilateral that you're given be $A = (x_1, y_1, 1), B = (x_2, y_2, 1), C = (x_3, y_3, 1) , D = (x_4, y_4, 1) $

Now similar to what you mentioned in your question, you connect the eye point to the four vertices, resulting in the projected vertices

$ A' = e + t_1 (A - e) $

$ B' = e + t_2 (B - e) $

$ C' = e + t_3 (C - e) $

$ D' = e + t_4 (D - e) $

The first condition that we have to impose on $A',B',C',D'$ is that

$ C' - B' = D' - A' $

making $A'B'C'D'$ a parallelogram.

The second condition is that

$ (B' - A') \cdot (D' - A') = 0 $

this ensures that the parallelogram is a rectangle.

The first condition becomes:

$ t_3 (C - e) - t_2 (B - e) = t_4 (D - e) - t_1 (A - e) \hspace{10pt}(1)$

The second condition becomes:

$ (t_2 (B - e) - t_1 (A - e) ) \cdot ( t_4 (D - e) - t_1 (A - e) ) = 0 \hspace{10pt} (2)$

Note that the third row of $(1)$ implies

$ t_3 - t_2 = t_4 - t_1 \hspace{10pt}(3) $

And from $(3)$, we cancel the terms involving the vector $e$ on both sides of $(1)$. And it becomes

$ t_3 C - t_2 B = t_4 D - t_1 A \hspace{10pt} (4) $

This equation involves three linear equations in $t_1, t_2, t_3, t_4 $

Its solution is

$ (t_1, t_2, t_3, t_4) = \lambda \mathbf{v} \hspace{10pt}(5) $

where $\lambda \in \mathbb{R}$ is an arbitrary non-zero scalar. and $\mathbf{v} \in \mathbb{R}^4$ is a known vector.

Equation $(2)$ is a quadratic equation in $ t_1, t_2, t_3, t_4, e_x, e_y$

Substituting $t_1, t_2, t_3, t_4$ from $(5)$ into $(2)$, and dividing by $\lambda^2$, it becomes,

$ (v_2 (B - e) - v_1 (A - e) ) \cdot ( v_4 (D - e) - v_1 (A - e) ) = 0 \hspace{10pt} (6)$

And $(6)$ is an equation of a circle. Re-arranging the terms in $(6)$, we get,

$ ( e ( v_1 - v_2) ) + v_2 B - v_1 A ) \cdot ( e (v_1 - v_4 ) + v_4 D - v_1 A ) = 0 $

Define

$ s_1 = v_1 - v_2 $

$ s_2 = v_1 - v_4 $

$ w_1 = v_2 B - v_1 A $

$ w_2 = v_4 D - v_1 A $

Then, the equation $(6)$ can be written compactly as,

$ (s_1 e + w_1)^T (s_2 e + w_2) = 0 \hspace{10pt}(7) $

so that,

$ s_1 s_2 e^T e + e^T ( s_1 w_2 + s_2 w_1 ) + w_1^T w_2 = 0 \hspace{10pt} (8) $

which is an equation of a circle in the $xy$ plane.

This determines all the possible positions of the eye of the observer that result in a projection that is a rectangle.

Example 1:

Suppose $A = (0, 0, 1) , B = (5, 1, 1), C = (6, 3, 1) , D = (1, 5, 1) $

Applying the above method, yields

$ \mathbf{v} = (\dfrac{4}{3}, 3, 8, 1) $

And the equation of the circle on which $e$ lies is

$ -\dfrac{5}{9} (e_x^2 + e_y^2) + \dfrac{10}{3} e_x + - \dfrac{22}{3} e_y + 29 \dfrac{4}{9} = 0 $

which becomes after dividing by the leading coefficient,

$ e_x^2 + e_y^2 - 6 e_x + 13.2 e_y - 53 = 0 $

The center of this circle is at

$ C_0 = ( 3 , - 6.6 ) $

And its radius is $ 16.22467 $

Taking the polar angle as $\dfrac{\pi}{4} $

Gives $ e = (10.26498, 0.664985, 0 )$

Taking $\lambda = 5 $ generates the pre-image (rectangle)

The image (quadrilateral) is plotted below in blue on the left, and the pre-image which is a rectangle is plotted on the right in red.

If we pick the polar angle on the circle to be, say, $\dfrac{2\pi}{3}$, then the observer eye position vector becomes

$ e = (-2.13712 , 2.297753 , 0) $

and we get the following pre-image. So the aspect ratio and dimensions of the rectangle will change depending on the location of the observer's eye.

Example 2:

As a special case, suppose the given image is a rectangle, in this case, equation $(8)$ becomes an identity, because we will have $s_1 s_2 = 0, ( s_1 w_2 + s_2 w_1 ) = 0 $, and $w_1^T w_2 = 0 $, hence the observer eye can be placed anywhere, and it will result in a pre-image that is a rectangle, as expected.