Find the homography matrix that maps a 2D image point to a 3D world point on a known plane

Question

Given a pinhole camera with known 3x4 calibration matrix $P$ how do we find the 3x3 homography matrix $H$ that can map 2D image points to 3D world points given that the 3D points are constrained to a known world plane $\pi$?

Hartley and Zissermann's Multiple View Geometry section 8.1.1 gives a simple case with the world plane $Z=0$.
$$x=PX=\begin{bmatrix}p1 & p2 & p3 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ 0 \\ 1\end{pmatrix} =\begin{bmatrix}p1 & p2 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ 1\end{pmatrix} $$
Here the 3x3 homography that maps world points to image points is given by $H=\begin{bmatrix}p1 & p2 & p4\end{bmatrix}$ and the inverse $H^{-1}$ maps homogeneous pixels to world points $$X=H^{-1}x$$ In my case the world points are constrained to a known plane that is parallel to $Z=0$ but offset some known amount $a$. I tried to solve: $$x=PX=\begin{bmatrix}p1 & p2 & p3 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ a \\ 1\end{pmatrix}$$ But I'm not sure how to proceed in this case because I can't invert the 3x4 matrix on the right. If it exists I'm looking for a solution that gives $H^{-1}$ with parameter $a$ so I don't have to invert a matrix for each point I want to map.

Answer 1

You can just change coordinates so that your parallel plane becomes the plane $Z=0$.

$P\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}=PA^{-1}\begin{bmatrix}X\\Y\\Z-h\\1\end{bmatrix}$ where $A=\begin{bmatrix}1&0&0&0\\0&1&0&0\\0&0&1&-h\\0&0&0&1\\\end{bmatrix}$.

I think then you can proceed as normal with $PA^{-1}$ by continuing with $Z-h=0$ (i.e. $Z=h$.)

That's the argument that first occurs to me anyway: change world coordinates to make your plane coincide with the $Z$ plane. I have not seen it written down, and I have not applied it, but that seems reasonable.

Answer 2

If you write

$$x=PX=\begin{bmatrix} |&|&|&|\\ p_1 & p_2 & p_3 & p_4 \\ |&|&|&| \end{bmatrix} \begin{pmatrix}X \\ Y \\ a \\ 1\end{pmatrix}$$

then the right hand side contains two variables and two constants. Since you are dealing with homogeneous coordinates, there is a third degree of freedom in scaling the whole vector. A result can be mapped back to an input of this form by employing these three degrees of freedom.

But in your notation, it looks like four independent elements. The reason why they have to be fixed is that the last two elements always have ratio $a:1$, for your chosen constant $a$. To make this clearer, you can combine these two constants into a single coordinate:

$$x=\begin{bmatrix} |&|&|\\ p_1 & p_2 & ap_3+p_4 \\ |&|&| \end{bmatrix} \begin{pmatrix}X \\ Y \\ 1\end{pmatrix}$$

Now the coefficient $a$ has moved to the matrix, which is now $3\times3$ and can be inverted (at least in general). If you compute the inverse, and then want to go back to your original coordinates, you get the first, second and fourth coordinate, and the third is $a$ times that fourth coordinate.