I have a 3D cube, 8 3D point cordinates, whose center is $(0,0,0)$, and is rotationally distorted. I want to calculate the angles, $R_x, R_y, R_z$ for rotation to rectificate the cube.
3D coordinates, $(x,y,z)$'s, of a sample cube:
p0 = (0,4482, 0,6137, 0,4153)
p1 = (0,1041, 0,6839, -0,5210)
p2 = (0,7758, -0,3119, 0,2255)
p3 = (0,4317, -0,2417, -0,7108)
p4 = (-0,4317, 0,2417, 0,7108)
p5 = (-0,7758, 0,3119, -0,2255)
p6 = (-0,1041, -0,6839, 0,5210)
p7 = (-0,4482, -0,6137, -0,4153)
which looks something like:
(the camera is at the point $(0,0,2)$ and looks to the point $(0,0,0)$)
I created 3 vectors using coordinates of the Cube, $V_x', V_y', V_z'$, $p_0 - p_1, p_0 - p_2, p_0-p_4$ these 3 vectors have some angles with the real world axis', $V_x, V_y, V_z$.
I know how to use dot product, however, using classical dot product does not give me the true rotation angles, $R_x, R_y, R_z$, and indeed I know the ordering of the rotation is important, (i.e., rotate X axis 30; rotate Y axis 40; is not the same as rotate Y axis 40; rotate X axis 30;).
My question is using the coordinate system of the cube, $V_x', V_y', V_z'$ and using the real coordinate system, $V_x, V_y, V_z$, $[1,0,0], [0,1,0],[0,0,1]$, how can I come up with the $R_x, R_y, R_z$ angles such that when I rotate all the points of the cube using $R_x, R_y, R_z$, respectively, the cube will be rectified (i.e., $V_x$ and $V_x'$, $V_y$ and $V_y'$, $V_z$ and $V_z'$ will be one after the other) ?
Or if my current approach is wrong, how can I come up with the angles, $R_x, R_y, R_z$, using $p_0, p_1, \ldots, p_7$ ?
Since I'm typing in a full answer, here's a little linear algebra for free. :)
If your original coordinate axes have been linearly transformed to the vectors $V_x'$, $V_y'$, and $V_z'$, the matrix which performed this transformation must have been $$R = \begin{bmatrix}V_x' & V_y' & V_z'\end{bmatrix},$$ i.e. the $3\times3$ matrix whose columns are $V_x'$, $V_y'$, and $V_z'$. (Try multiplying this matrix with the three axis-aligned unit vectors to see why.) The matrix you need to undo that transformation is its inverse, $R^{-1}$. Since in your case $R$ is a rotation matrix, $V_x'$, $V_y'$ and $V_z'$ must be of unit length and mutually orthogonal. This also means that $R^{-1}$ is the same as the transpose, $R^T$, which is the matrix whose rows are $V_x'$, $V_y'$, and $V_z'$: $$R^{-1} = R^T = \begin{bmatrix}V_x'^T \\ V_y'^T \\ V_z'^T\end{bmatrix}.$$
So now you have a rotation matrix $R^{T}$ which transforms your cube back to being axis-aligned. But you want the corresponding Euler angles, say in the order $ZXZ$. Wikipedia gives the matrix corresponding to rotation angles $\theta_1$, $\theta_2$, $\theta_3$ as $$A = R_Z(\theta_1) R_X(\theta_2) R_Z(\theta_3) = \begin{bmatrix} c_1 c_3 - c_2 s_1 s_3 & - c_1 s_3 - c_2 c_3 s_1 & s_1 s_2 \\ c_3 s_1 + c_1 c_2 s_3 & c_1 c_2 c_3 - s_1 s_3 & - c_1 s_2 \\ s_2 s_3 & c_3 s_2 & c_2 \end{bmatrix},$$ where $c_i$ and $s_i$ mean $\cos\theta_i$ and $\sin\theta_i$ respectively. So we can see that $$\begin{align} a_{13}/a_{23} &= -\tan\theta_1, \\ a_{33} &= \cos\theta_2, \\ a_{31}/a_{32} &= \tan\theta_3. \\ \end{align}$$ Of course, $A$ is nothing but $R^T$, so we already know the values of its entries, and we can recover $\theta_1$, $\theta_2$, and $\theta_3$.