3D projection coordinates onto 2D plane to determine transformation matrix?

Question

I'm not sure if there is an actual solution to this problem or not, but thought I would give it a shot here to see if anyone has any ideas. So here goes:

I basically have three vertices of a rigid triangle with known 3D coordinates. The vertices are projected onto a 2D plane (by projection, I mean that each vertex would basically have a fixed line drawn from it to the 2D plane, and that "line" would also stay rigid to the triangle so that the lines would move along with the triangle if it is transformed), in which I also know the 2D coordinates. A transformation matrix is applied to the original three points (can be a combination of rotation and translation) and I now know the new 2D projection coordinates.

Is it possible to obtain either the unknown transformation matrix or the new coordinates? Any ideas are much appreciated. Thanks!

Answer 1

Here's a way of doing what I think you want to do.

First, find two perpendicular unit-vectors that are parallel to the plane (i.e. perpendicular to the normal vector). You might want to use Gram-Schmidt here. Call the vectors $u_1$ and $u_2$. These are the "axes" along the plane you're projecting onto.

Any point on the plane can be written in the form $a_1u_1 + a_2u_2$ for some coordinates $(a_1,a_2)$. To find the coordinates of the orthogonal projection of a vector $x$ onto your plane, calculate $$ a_1 = u_1^Tx\\ a_2 = u_2^Tx $$ Or, all together: $$ \pmatrix{a_1\\a_2} = \pmatrix{u_1^T\\u_2^T} \pmatrix{x_1\\x_2\\x_3} $$

Answer 2

In general, the answer to your problem is no, it is not possible to do so. There are however variants that are possible if we know more about the transformation.

Both I get into detail, some notation and two observations:

Notation: We know 3 points, $a$,$b$,$c$ and a projection $P$ onto a 2d plane through the origin (so $P$ is a matrix with $P^2=P$ and $P$ is of rank 2). There is an unknown matrix $M$, but we know the three points $PMa, PMb$ an $PMc$.

1. I assume your original points $a$,$b$,$c$ are linearly independent, otherwise (for example if one of them is (0,0,0)) it will clearly not work.

2. It does not matter if we recover the 3d image points $Ma,Mb$ and $Mc$ or the transformation matrix. If we have the matrix we can apply it to $a,b,c$ in order to get the points. If we have the points, we can recover the matrix, since $a,b,c$ form a basis and any matrix is uniquely given by its image in any basis (see footnote 1).

Now to the actual problem. By a simple heuristic, we know three 2d coordinates, that is 6 independent values (the initial points $a,b,c$ do not count, see footnote 2), but we want to recover all 9 independent entries of $M$. There is obviously a mismatch here. This is no proof, but it tells us to search for a counterexample:

The problem essentialy is that we do not know the height the points $Ma,Mb$ and $Mc$ have above the plane. In fact let $v\neq 0$ be in the kernel of $P$, that is $Pv = 0$ (such a $v$ exists, since $P$ is of rank 2). Let $A$ be a matrix whose collumns are all multiples of $v$. Then the image of $A$ is the kernel of $P$ and we have $$ P(M+A) x = PMx+PAx = PMx$$ for any $x$. So using only the data given we cannot distinguish $M$ and $M+A$ as transformations, thus the answer is no, it is not possible.

As I said in the beginning, however there variants, for which this is possible. For example, if we know that $M$ is a rotation, we search for fewer independent values, so it should be possible. As said in the footnotes, we know the matrix $PM$. Along the axis of rotation we know that $Mx=x$ and thus $PMx=Px$, which is a linear equation with a 1d space of solutions (except when $M$ is the identity, but that is exactly the case if $PM=P$, so we can exclude this) which then give us this axis.

Let us call the axis $v$. Now we have two cases:

1. The axis of rotation does not lie in the plane on which we project. Then $P$ restricted to vectors from the plane of rotation $v^\perp$ is invertible, so we can recover how $M$ transforms the plane $v^\perp$ and thus recover $M$.

2. The axis of rotation lies in the plane on which we project. Then there is a vector $x$ in $v^\perp$ such that $PMx=0$, since $x$ is rotated to a vector perpendicular to the plane. But then we can find $x$ and recover the angle of rotation this way, also finding $M$.

Footnotes:

1.More detail, by linear independence, there are $r,s,t$ such that $(1,0,0)^T=r a+sb+tc$, but then $M(1,0,0)^T = rM a+sMb+tMc$ so we recover the first collumn of $M$. Then continue in the same way for $(0,1,0)$ and $(0,0,1)$.

2. The points $a,b,c$ do not contain any information about $M$, they only serve to determine the images. However note that using the same idea as in footnote 1, we can actually recover the matrix $PM$ from the initial information. We cannot get $M$ from this, since $P$ is of rank 2 and thus not invertible.