Image projection, 3D, Planar homography

Question

I am currently studying image stiching and I am reading this very intresting paper form Microsoft: https://www.microsoft.com/en-us/research/wp-content/uploads/2004/10/tr-2004-92.pdf

I just started, so I still have some difficulties ... maybe you can help me ?

On page 4 and 5, planar homography is described. I don't understand this section. Basically, what I have understood, is that you have something like a wall. Now you make a picture of it from two different positions. Finally you would like to know the transforation matrix to go from image A to image B using H10.

Can someone explain me in simple words, how this is achieved ?

Thank you for any help !!!

Answer 1

It is a projection, you have the center of projection, an object and the plane where you project it, something like this
Your object is in plain z=1 - [possible coords. (x,y,1)] and you shoot rays from (0,0,0) all around...
Objects shadow falls on a new plane(blue wall) with origin and axises on its own $O'(\vec X',\vec Y'), O'(o_1,o_2,o_3) X'=(x_1,x_2,x_3)Y'=(y_1,y_2,y_3)$ so each point in the new plane has coordinates $(o_1+x'x_1+y'y_1,o_2+x'x_2+y'y_2,o_3+x'x_3+y'y_3)$ where x' and y' are coordinates in the new plane

To get (x,y,1) you have to divide it by $o_3+x'x_3+y'y_3$ cos they are on the same ray
so $x=\frac{o_1+x'x_1+y'y_1}{o_3+x'x_3+y'y_3}, y=\frac{o_2+x'x_2+y'y_2}{o_3+x'x_3+y'y_3}$
As you see it already looks like the given formula, so the parameters from the matrix are the coordinates of origin vector and axis vectors of the projection plane

Actually not quite same, cos x, y and x',y' are swapped but you see the point. You don't need all the details ;)

Answer 2

Lets start with general definition of the transformation points in 3D plane from one camera to camera':

$$ \mathbf{X'} = \tilde{H}\mathbf{X} $$

In case of image planes using homogeneous coordinates:

$$ λ'\mathbf{\tilde{x}'} = \mathbf{X'}, λ\mathbf{\tilde{x}} = \mathbf{X}, \Rightarrow λ'\mathbf{\tilde{x}'} = \tilde{H}λ\mathbf{\tilde{x}} $$

This means that $\mathbf{\tilde{x}'}$ is equal to $\tilde{H}\mathbf{\tilde{x}}$ up to scale. If it is known that some points all lie in a plane in an image, the image can be rectified directly without needing to recover 3D coordinates.

From the paper follows the next:

$$ \mathbf{\tilde{x}'} \sim \tilde{H} \mathbf{\tilde{x}} \Rightarrow \begin{bmatrix} x' \\ y' \\ z' \end{bmatrix} = \begin{bmatrix} h_{00} & h_{01} & h_{02} \\ h_{10} & h_{11} & h_{22} \\ h_{20} & h_{21} & h_{22} \\ \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} $$

In case of inhomogenous coordinates ($x' = x/z$ and $y' = y/z$):

$$ x' = \frac{h_{00}x + h_{01}y + h_{02}z}{h_{20}x + h_{21}y + h_{22}z}; y' = \frac{h_{10}x + h_{11}y + h_{12}z}{h_{20}x + h_{21}y + h_{22}z} $$

In 2D plane z-coordinates have value 1 (without loss of generality). Thus we have formulas from the paper:

$$ x' = \frac{h_{00}x + h_{01}y + h_{02}}{h_{20}x + h_{21}y + h_{22}}; y' = \frac{h_{10}x + h_{11}y + h_{12}}{h_{20}x + h_{21}y + h_{22}} $$

Also you can find the detailed explanation of the planar homography in Lecture 16: Planar Homographies.