Dear Math Stackexchange community, how can I compute $R$ rotation matrix in this equation ($P=Rp+t$)? (Please see the figure. More detailed explanation is under the figure.)
The equation is for the transformation of $p=p_1,p_2,p_3$ points. $p$ is in Oxyz (local) reference frame. $p$ consists of 3 points of $p_1,p_2,p_3$. And $p_1,p_2,p_3$ each has 3 coordinates of $x,y,z$. Therefore p is 3x3 matrix. $p=\begin{bmatrix} p_{1x} & p_{2x} & p_{3x} \\ p_{1y} & p_{2y} & p_{3y} \\ p_{1z} & p_{2z} & p_{3z} \end{bmatrix}$
$t$ is the distance between Oxyz(local) and OXYZ(global) reference frame. $t=\begin{bmatrix} t_x \\ t_y \\ t_z \end{bmatrix}$. To be able to do matrix operations, we can take $t$ as $t=\begin{bmatrix} t_x&t_x&t_x \\ t_y&t_y&t_y \\ t_z&t_z&t_z \end{bmatrix}$
$P$ is the coordinates of p_1, p_2, and p_3 in OXYZ global reference frame. $P$ is a 3x3 matrix. $P=\begin{bmatrix} P_{1x} & P_{2x} & P_{3x} \\ P_{1y} & P_{2y} & P_{3y} \\ P_{1z} & P_{2z} & P_{3z} \end{bmatrix}$
$R$ is the (orthogonal) rotation matrix. $R=\begin{bmatrix} r_{11}&r_{12}&r_{13} \\ r_{21}&r_{22}&r_{23} \\ r_{31}&r_{32}&r_{33} \end{bmatrix}$
Therefore, all in all: $P=Rp+t$
$\begin{bmatrix} P_{1x} & P_{2x} & P_{3x} \\ P_{1y} & P_{2y} & P_{3y} \\ P_{1z} & P_{2z} & P_{3z} \end{bmatrix} = \begin{bmatrix} r_{11}&r_{12}&r_{13} \\ r_{21}&r_{22}&r_{23} \\ r_{31}&r_{32}&r_{33} \end{bmatrix}.\begin{bmatrix} p_{1x} & p_{2x} & p_{3x} \\ p_{1y} & p_{2y} & p_{3y} \\ p_{1z} & p_{2z} & p_{3z} \end{bmatrix}+\begin{bmatrix} t_x&t_x&t_x \\ t_y&t_y&t_y \\ t_z&t_z&t_z \end{bmatrix}$
We can derive the following form of the equation of $P=Rp+t$
$(P-t)=R.p$
$(P-t).p^{-1}=R$
$p$ is an arbitrary matrix, and not always be taken of its inverse. I can't take the inverse of $p$, if last raw of $p$ is zero.
Let's look at an example to make it clear: $P=Rp+t$
$p_1=(1,1,0)$ , $p_2=(2,4,0)$ , $p_3=(5,10,0)$ is the coordinates of three points. So in matrix form: $p=\begin{bmatrix} 1 & 2 & 5 \\ 1 & 4 & 10 \\ 0 & 0 & 0 \end{bmatrix}$
$t=\begin{bmatrix} 0 \\ 0 \\ 50 \end{bmatrix}$
$\alpha=10^\circ $ (rotation around the x axis)
$\beta=15^\circ $ (rotation around the y axis)
$\gamma=20^\circ $ (rotation around the z axis)
$R_x=\begin{bmatrix} 1 & 0 & 0 \\ 0 & cos(\alpha) & -sin(\alpha) \\ 0 & sin(\alpha) & cos(\alpha) \end{bmatrix}$
$R_y=\begin{bmatrix} cos(\beta) & 0 & sin(\beta) \\ 0 & 1 & 0 \\ -sin(\beta) & 0 & cos(\beta) \end{bmatrix}$
$R_x=\begin{bmatrix} cos(\gamma) & -sin(\gamma) & 0 \\ sin(\gamma) & cos(\gamma) & 0 \\ 0 & 0 & 1 \end{bmatrix}$
$R=R_z.R_y.R_x$
Therefore, for $\alpha=10^\circ $, $\beta=15^\circ $, $\gamma=20^\circ $, the rotation matrix is :
$R=\begin{bmatrix} 0.9077 & -0.2946 & 0.2989 \\ 0.3304 & 0.9408 & -0.0760 \\ -0.2588 & 0.1677 & 0.9513 \end{bmatrix}$
$P=R.p+t$
$P=$$\begin{bmatrix} 0.9077 & -0.2946 & 0.2989 \\ 0.3304 & 0.9408 & -0.0760 \\ -0.2588 & 0.1677 & 0.9513 \end{bmatrix}$$.\begin{bmatrix} 1 & 2 & 5 \\ 1 & 4 & 10 \\ 0 & 0 & 0 \end{bmatrix}$ $+\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 50 & 50 & 50 \end{bmatrix}$
$P=\begin{bmatrix} 0.6131 & 0.6370 & 1.5925 \\ 1.2712 & 4.4239 & 11.0597 \\ 49.9089 & 50.1533 & 50.3832 \end{bmatrix}$
$p_1=(0.6131,1.2712,49.90)$,$p_2=(0.6370,4.4239,50.1533)$,$p_3=(1.5925,11.0597,50.3832)$.
Now let's take opposite example with the same values to find $R$, with the values of $P,p,t$ as known.
$P=R.p+t$
$\begin{bmatrix} 0.6131 & 0.6370 & 1.5925 \\ 1.2712 & 4.4239 & 11.0597 \\ 49.9089 & 50.1533 & 50.3832 \end{bmatrix}=R.$ $\begin{bmatrix} 1 & 2 & 5 \\ 1 & 4 & 10 \\ 0 & 0 & 0 \end{bmatrix}+$ $\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 50 & 50 & 50 \end{bmatrix}$
Could you find $R$ here in the above equation?
To repeat the formulation:
$P=Rp+t$
$(P-t)=Rp$
$(P-t).p^{-1}=R.p.p^{-1}$
$(P-t).p^{-1}=R$
For the inverse of $p$ (where $p$ is $\begin{bmatrix} 1 & 2 & 5 \\ 1 & 4 & 10 \\ 0 & 0 & 0 \end{bmatrix}$ ), I can't take the inverse of $p$.
Here I am stuck. It is interesting for me, because for the condition that $R,p,t$ is known, $P$ can be calculated(9 unknown elements in 3x3 matrix of $P$). However, for the condition that $P,p,t$ is known, I am having problem to compute $R$ (9 unknown elements in 3x3 matrix of $R$). I should be able to compute the equivalent $R$ matrix for that inverse problem.
Do you have any idea about how to find $R$ rotation matrix? Any suggestion would be appreciated.
Thanks in advance.

Firstly, I think the extra gymnastics with $t$ is a distraction, so I will state the problem as:
$\mathbf{Q}=\left[\mathbf{q}_1\,\dots,\mathbf{q}_k\right]=\mathbf{P}-\mathbf{t}=\mathbf{R}.\boldsymbol{\mathcal{P}}=\mathbf{R}.\left[\mathbf{p}_1\,\dots\,\mathbf{p}_k\right]$
Where $\mathbf{Q}$ is some $3\times k$ real-valued matrix, which we can also view as $k$ column vectors $\mathbf{q}_{1,\dots k}$. $\mathbf{R}$ is a real-valued rotation matrix, which means it is orthogonal ($\mathbf{R}^T\mathbf{R}=\mathbf{Id}=identity$) and has determinant of $det\left[\mathbf{R}\right]=+1$. Finally, matrix $\boldsymbol{\mathcal{P}}$ is made up from 3d real-valued column vectors $\mathbf{p}_{1,\dots k}$. The aim is to determine the rotation matrix.
Here, for simplicity, I will work with linearly independent sets of $\mathbf{p}$-vectors, so $k=\mbox{rank}\left(\mathbf{Q}\right)=\mbox{rank}\left(\boldsymbol{\mathcal{P}}\right)$
$k=\mbox{rank}\boldsymbol{\mathcal{P}}=3$
If you can pick three linearly independent vectors $\mathbf{p}_{1,2,3}$ you can have a unique solution for $\mathbf{R}$ given $\mathbf{Q}$ (by inverting $\boldsymbol{\mathcal{P}}$). If you cannot such unique solution does not exist.
Use
rankfunction to check whether you have independent $\mathbf{p}$-vectors$k=\mbox{rank}\boldsymbol{\mathcal{P}}=2$
If you only have two linearly independent vectors, I think the best you can in principle get two possible solutions, of which both fit the data. In some special cases you will still be able to get the unique solution.
Use Gram-Schmidt process to extract two orthonormal vectors $\mathbf{\hat{n}}_{1,2}$ which can be expressed as $\mathbf{\hat{n}}_{1,2}=\alpha_{1,2}\mathbf{p}_1+\beta_{1,2}\mathbf{p}_2$.
There will be two further orthonormal vectors $\mathbf{\hat{q}}_{1,2}=\alpha_{1,2}\mathbf{q}_1+\beta_{1,2}\mathbf{q}_2$.
Now try to find $\psi$ that minimizes: $s=\left|\left(\cos\psi\,\mathbf{\hat{q}_1+\sin\psi\,\mathbf{\hat{q}}_2}\right)-\left(\cos\psi\,\mathbf{\hat{n}_1+\sin\psi\,\mathbf{\hat{n}}_2}\right)\right|$, this will give you the projection of the axis around which matrix $\mathbf{R}$ causes the rotation. Let the actual rotation axis be $\boldsymbol{{\hat{\omega}}}$, and its projection into plane spanned by $\mathbf{\hat{n}}_{1,2}$ be $\boldsymbol{{\omega}}_\mathcal{P}$. The logic here is simple if $\boldsymbol{\hat{\omega}}=\left(\cos\psi\,\mathbf{\hat{n}_1+\sin\psi\,\mathbf{\hat{n}}_2}\right)$ then rotating such vector will not change it at all, by definition.
Two interesting cases here are when $s=const$ for all $\psi$. Then your rotation matrix's axis is perpendicular to the plane spanned by $\mathbf{\hat{n}}_{1,2}$, and you can make the problem 2d (i.e. make $\mathbf{R}$ 2d) and find exact solution.
Another interesting case is when $s=0$ or very close to 0, say 1e-3, for some $\psi$. Then you found a way to express the rotation axis in terms of your orthonormal vectors: $\boldsymbol{\hat{\omega}}=\left(\cos\psi\,\mathbf{\hat{n}_1+\sin\psi\,\mathbf{\hat{n}}_2}\right)$. I think you will be able to work out the rotation angle of $\mathbf{R}$ in this case by looking at what happens with $-\sin\psi\mathbf{\hat{n}}_1+\cos\psi\mathbf{\hat{n}}_2$, i.e. vector that is perpendicular to rotation axis.
In general, if you found $\boldsymbol{\omega}_\mathcal{P}$ you can extend it to full rotation axis by adding to it a component of vector perpendicular to the plane spanned by $\mathbf{\hat{n}}_{1,2}$, $\mathbf{\hat{m}}=\mathbf{\hat{n}}_1\times \mathbf{\hat{n}}_2$. So
$$ \boldsymbol{\hat{\omega}}=\frac{\boldsymbol{\omega}_{\mathcal{P}}+f\mathbf{\hat{n}}_1\times \mathbf{\hat{n}}_2}{\left|\boldsymbol{\omega}_{\mathcal{P}}+f\mathbf{\hat{n}}_1\times \mathbf{\hat{n}}_2\right|} $$
There will be ambiguity in choosing the sign of $f$ - but that's the non-uniqueness for you. Once you have the rotation axis, you should be able to reduce the problem to 2d again, and solve it exactly (up to sign of $f$)
$k=\mbox{rank}\boldsymbol{\mathcal{P}}=1$
Not sure there is much you can do here.