Restore the original 3D points from projected 2D polygon knowing local positions of vertices

Question

Say you're given a 3x4 projection matrix P, and projected points on the 2D plane. And you also know, for a given T, 3D points of the rectangle can be represented as T+X1, T+X2 etc.

Now, knowing that every point in the projected 2D plane maps to a single line in the 3D space, it's obvious that we can't recover position a single vertex in 3D space. But, since in this case we know the relations between each vertex, I think we should be able to. But I don't know how to actually do it.

For now, assume the rectangle is parallel to the 2D projection plane, it's obvious that there's only one rectangle in 3D space that correspons to the projected rectangle. Since as the lines diverge, dimensions of the rectangle change. And this is just the case where z (in-plane) components of X1, X2 .. X4 are 0. So I assume any arbitrary configuration won't be much different.

Here's an illustration

Edit:

I tried to pick a T such that $$ T = A_0 + Null(P) * t $$Where A0 is a particular soln for point y0. And then I formed 3 equations for other points such that T calculated with given t, would satisfy other points as well. $$ Mi = P [I;0, Xi;1] [I;0, X0;1]^{-1} $$ $$ M Nul(P) t = y_i-M A0 $$ The interesting thing is that, equations by themselves are still underdetermined because M is of 3x4. But solutions of the equations form an overdetermined system. Which makes sense as your X1, ... X4 values may not actually form a trapezoid which can then be project onto 2D plane as a rectangle.

So for now, any numerical soln. I tried for the eq. above worked fine. But I'd be glad to find a more rigid and mathematical way of doing this.

Answer 1

Here's my attempt to solve this problem / draft solution:

1. Projection matrix P is basically a function $f :: (x,y,z) \rightarrow (x_1,y_1)$, with $f(p) = P(p)$, using matrix multiplication. We can use it to project an unknown line to a single 2d point, i.e. $g :: (x(z),y(z),z) \rightarrow (x_1,y_1)$
2. Inverse of the projection matrix gives a line in 3d space: $g^{-1}([x_1,y_1]^{T})(z) = P^{-1}[x_1,y_1,z]^{T}$. (note the $z$ is special)
3. Taking $X_1, X_2, X_3, X_4$ into account and we can create a distance function from point $C(C_x,C_y,C_z)$, i.e. $$d(x,y,z)=\sqrt{(x-C_x)^2+(y-C_y)^2+(z-C_z)^2}$$. For different points: $d(P_1) = X_1$. $d(P_2) = X_2$. $d(P_3)=X_3$. $d(P_4)=X_4.$
4. Unfortunately we don't know location of point C, but nice thing is that each of the equation is sharing the point C.
5. Then there's the projected 2d points: p_1,p_2,p_3,p_4. This (together with (2)) allows us to calculate 4 lines in 3d space. They should be in form $l(z) = (x(z),y(z),z)$.
6. Calculating point $C$ can be done based on the 4 lines and $X_1,X_2,X_3,X_4$.
7. Then we can use our distance function 4 times: $d(l_n(z_n))=X_n$ to get 4 equations.
8. This equation can be solved to find value for $z_1,z_2,z_3,z_4$.
9. Then $l_n(z_n)$ can be used to calculate the 3d positions required.

(getting the actual values via this solution approach seems quite difficult, given all the equation solving required -- so that part is left as excersize for the reader -- maybe there's easier way to get a solution?)