I’m working with transformation matrices 4 x 4 (using homogeneous coordinates) Those transformations can include rotations (Euler angles), translations, scales, skews, and so on
What i’ve got :
A box positioned in space by a transformation matrix, in which points are positioned according its basis. I know the transformation matrix to express those points in space coordinates. A plane, defined by 2 vectors and a point. A transformation matrix defining the orthogonal projection of box’s points on this plane, in space coordinates. A rectangle that is a face of the box, on which points are positioned according 2D coordinates (we define randomly the 2d origin as the “bottom left” corner, looking at the rectangle parallely to its normal)
I’m looking for the 2D transformation matrix 3x3 to apply in order to “deform” the rectangle so that its shape will be the same as its projection on the plane.
Concretely, i’m working with boxes positioned in space, and possibly deformed by skews and others. On a face of these box, there is a vectorial drawing. I want to “flatten” the drawing of one box to a plane. The drawing (using SVG) of the face will be done in a group on which the transformation matrix i’m looking for will be applied.
There’s quite a bit that you’ve left unspecified, so this will be the outline of a solution instead of a complete one.
You’ve got a lot of what you need already: the box transformation $B$ and the projection $P$. In addition, you’ll need a “face map” $F$, which defines how the image rectangle is mapped onto the face of the untransformed box, and finally, the coordinate transformation $C$ that maps from space coordinates to image plane coordinates. The transformation you seek is then a plane homography that’s the composition of all of these transformations, $H=CPBF$. To construct the missing matrices, I’ll rely heavily on the fact that the columns of a transformation matrix are the images of the basis.
In the following, I’ll use upper-case bold letters for homogeneous coordinate vectors in $\mathbb R^3$ (really $\mathbb{RP}^3$) and lower-case for homogeneous coordinate vectors in $\mathbb R^2$. A tilde indicates the corresponding inhomogeneous (Cartesian) coordinate vector. These are all column vectors, and are transformed via left-multiplication by an appropriate matrix.
For the “face map” $F$, I’ll assume that the initial rectangle is axis-aligned and has its lower-left corner at the origin. The description you gave of how this rectangle is mapped to a box face is ambiguous: leaving aside the question of whether the surface normal points inward or outward, the view can be rotated about this axis to make any corner of the face the “lower left.” However, let’s say that the vertex of the box that corresponds to the rectangle’s lower-left corner is $\mathbf{LL}$ and that $\mathbf{FH}$ and $\mathbf{FV}$ are the unit direction vectors that define the edges of the box that correspond to the horizontal and vertical edges of the rectangle, respectively. Then the matrix of the face map is simply $$F=\begin{bmatrix}\mathbf{FH}&\mathbf{FV}&\mathbf{LL}\end{bmatrix}.$$ This is a $4\times3$ homogeneous matrix.
At the other end of the cascade, we need a transformation from world coordinates into image plane coordinates, which requires that we first specify a coordinate system. Choose any point $\mathbf P_0$ on the image plane as the origin of this coordinate system. Since the image plane is specified by a point and a linearly-independent pair of vectors $\mathbf V_1$ and $\mathbf V_2$ parallel to the plane, you can use that point as the origin. We will translate the origin to this point.
Next, we need a pair or orthogonal unit vectors parallel to the image plane. You haven’t said whether or not $\mathbf V_1$ and $\mathbf V_2$ are already an orthonormal pair. If they are, you can use them directly and skip computing their cross product (see why below). If not, we’ll use them to generate an orthonormal set: Let $\tilde{\mathbf U}=\tilde{\mathbf V}_1/\|\tilde{\mathbf V}_1\|$ (i.e., normalize it if necessary), $\tilde{\mathbf N}$ to $\tilde{\mathbf V}_1\times\tilde{\mathbf V}_2$, also normalized, and finally, $\tilde{\mathbf V}=\tilde{\mathbf N}\times\tilde{\mathbf U}$. These vectors form an orthonormal basis with $\tilde{\mathbf N}$ perpendicular to the image plane. There are, of course, other ways to construct an orthonormal basis for the image plane, many of which you can find on this site, but this one uses the given data. (You can extract the necessary information from the projection matrix, for which see the addendum at the end.)
We construct a $3\times3$ rotation matrix that maps to this basis: $$R=\begin{bmatrix}\tilde{\mathbf U}^T\\\tilde{\mathbf V}^T\\\tilde{\mathbf N}^T\end{bmatrix}.$$ Note that the basis vectors that we just constructed are the rows of this matrix. We now have all of the pieces we need to construct the final mapping $C$: $$C=\begin{bmatrix}1&0&0&0\\0&1&0&0\\0&0&0&1\end{bmatrix}\left[\begin{array}{c|c}R&\mathbf 0\\ \hline \mathbf 0^T&1\end{array}\right]\left[\begin{array}{c|c}I & -\tilde{\mathbf P}_0 \\ \hline \mathbf 0^T & 1 \end{array}\right].$$ From right to left, the component matrices translate to $\mathbf P_0$, rotate, and then discard the third coordinate. We can easily compute the resulting matrix by using the fact that each element of a matrix product is a dot product of a row and column: $$C=\left[\begin{array}{c|c} \tilde{\mathbf U}^T & -\tilde{\mathbf U}^T\tilde{\mathbf C} \\ \tilde{\mathbf V}^T & -\tilde{\mathbf V}^T\tilde{\mathbf C} \\ \hline \mathbf 0^T & 1 \end{array}\right].$$ $\mathbf N$ doesn’t appear in the result, so if the vectors that define the image plane are already orthonormal, there’s no need to compute this image plane normal. (As a mildly interesting side note, the rows of this matrix are the planes through $\mathbf C$ with normals $\mathbf U$ and $\mathbf V$, respectively, together with the plane at infinity.)
Now, a coordinate is essentially a scalar projection onto a basis vector, so the orthogonal projection $P$ onto the image plane is redundant with $C$ and we can cut it out of the cascade. Therefore, when all is said and done the homography you’re looking for can be constructed as the product of three matrices: $H=CBF$. If $B$ is an affine transformation, then so is $H$. If you’re only interested in how the original face image is warped, you can extract the $2\times2$ linear part of $H$ and ignore the translation. (This is equivalent to choosing the projected image of the rectangle’s lower-left corner as the origin of the image plane coordinates.) As well, there are other interesting choices that can be made for the image plane basis. For instance, the first column of $PBF$ is the image of the original rectangle’s $x$-axis, so is the direction vector of the projected lower edge of the rectangle.
There’s one other consideration: The normal $\mathbf N$ defines the side from which we’re viewing the image plane. Using this same normal for all of the faces will result in some of the projected images being reversed. If you don’t want this to happen, you can instead use $-\mathbf N$ when constructing the image plane basis for those faces. Examining the sign of the dot product of the face normal with $\tilde{\mathbf N}$ will allow you to detect when this will need to be done.
If you have access to the orthogonal projection matrix $P$, you can extract the parameters of the image plane from it. The last column is the image of the world origin, so is a point $\mathbf P_0$ on the image plane. Any linearly independent pair of the remaining three columns can serve as $\mathbf V_1$ and $\mathbf V_2$, the vectors parallel to the plane.