Hope somebody can help me out of this, I did not find an answer to this one, so here we go:
Given a Coordinate Frame F (x,y,z) and a rectangle rect (Points and Vectors are given) in 3D space, i am searching for a Rotation Matrix M (or Transformation) so that the rectangle's x- and y-Vectors align with the given coordinate frame x- and y-axes (directions should be also the same). For a better understanding i drew a small graphic. This should be working for random rectangles in 3D space.
Thanks for your help!
Let's say your rectangle "origin" corner is at $\vec{o}$, "positive x" corner is at $\vec{u}$, and the "positive y" corner is at $\vec{v}$. This means the fourth corner is at $\vec{o} + \left( \vec{u} - \vec{o} \right) + \left( \vec{v} - \vec{o} \right) = \vec{u} + \vec{v} - \vec{o}$.
(Note that I assume a $3 \times 3$ rotation matrix $\mathbf{R}$ applied to a column vector $\vec{p}$ so that $\vec{p}$ rotates to $\vec{p}'$ is calculated as $\vec{p}' = \mathbf{R} \vec{p}$.)
Let $\hat{e}_1$ be an unit vector parallel to the rectangle $x$ side, $$\hat{e}_1 = (x_1, y_1, z_1) = \frac{\vec{u} - \vec{o}}{\left\lVert \vec{u} - \vec{o} \right\rVert}$$ and $\hat{e}_2$ be an unit vector parallel to the rectangle $y$ side, $$\hat{e}_2 = (x_2, y_2, z_2) = \frac{\vec{v} - \vec{o}}{\left\lVert \vec{v} - \vec{o} \right\rVert}$$ By the definition of a rectangle, $\hat{e}_1$ and $\hat{e}_2$ are perpendicular to each other, and being unit vectors, they are suitable for a new orthonormal basis (new coordinate system). The third basis vector is $$\hat{e}_3 = (x_3, y_3, z_3) = \hat{e}_1 \times \hat{e}_2$$
If we form a $3\times 3$ matrix from column vectors $\hat{e}_1$, $\hat{e}_2$, and $\hat{e}_3$, we get a rotation matrix that rotates the axes to parallel your rectangle. Obviously, we need its inverse. Fortunately, pure rotation matrices are orthogonal, and their inverse is the same as their transpose. Thus, we just need to form the rotation matrix from the transposed vectors, each row corresponding to one vector (because we are looking for a matrix that rotates the vectors parallel to coordinate axes):
$$\mathbf{R} = \left [ \begin{matrix} x_1 & y_1 & z_1 \\ x_2 & y_2 & z_2 \\ x_3 & y_3 & z_3 \end{matrix} \right ]$$
If you want to move $\vec{o}$ to origin, apply a translation by $-\vec{o}$ before the rotation. If you instead wish to apply the translation after the rotation (as most computer graphics libraries do), just rotate the translation too. That is, a translation by $-\vec{o}$ before rotation is equivalent to translation by $-\mathbf{R}\vec{o}$ after rotation.
If you want to keep $\vec{o}$ stationary (rotating the coordinate system around it), you need to apply a translation by $-\vec{o}$ before the rotation, and by $\vec{o}$ after the rotation. You can combine both into a translation after rotation; it is $$\vec{o} - \mathbf{R}\vec{o}$$ Simply put, each point $\vec{p}$ is rotated by $\mathbf{R}$ around point $\vec{o}$ using $$\vec{p}' = \mathbf{R} \vec{p} + \vec{t}$$ where $$\vec{t} = \vec{o} - \mathbf{R}\vec{o}$$
(It is often useful to remember that translation before rotation $\vec{b}$ and translation after rotation $a$ are easily converted, $$\begin{align}\vec{b} &= \mathbf{R}^T \vec{a} \\ \vec{a} &= \mathbf{R} \vec{b}\end{align}$$ and that translations can be just summed together, as long as they are all either before, or all after, the rotation.)
Very often computer graphics packages use $4 \times 4$ matrices to represent all kinds transformations, including rotations and translations (after rotation), with 3D vectors having an implicit fourth component with always a value of $1$. These matrices look like $$\mathbf{M} = \left [ \begin{matrix} x_X & x_Y & x_Z & x_T \\ y_X & y_Y & y_Z & y_T \\ z_X & z_Y & z_Z & z_T \\ 0 & 0 & 0 & 1 \end{matrix} \right ]$$ $x$ axis will be parallel to $(x_X, y_X, z_X)$ after the rotation, $y$ axis will be parallel to $(x_Y, y_Y, z_Y)$, and $z$ axis to $(x_Z, y_Z, z_Z)$. After the rotation, a translation by $(x_T, y_T, z_T)$ is applied (essentially, added to the resulting vectors). The last four components, the bottom row of the matrix, may be implicit too (not saved or accessed by the library).