I know there is a closed-form formula to bring together two sets of 3D points. Is there also a closed-form formula when one set of points is 2D and the other is projected?
More formaly, let $\left(\mathbf{x}_i\right)_n\in\mathbf{R}^{3\times n}$ and $\left(\mathbf{y}_i\right)_n\in\mathbf{R}^{2\times n}$ be two sequences of length $n$ of resp. 3D and 2D points.
I am looking for a closed-form solution for the 3D rotation $R^*\in SO(3)$ and translation $\mathbf{t}^*\in\mathbf{R}^3$ so that \begin{eqnarray} (R^*, \mathbf{t}^*) = \arg\min_{R, t} \sum_{i=1}^n \left\|\mathbf{y}_i - (PR\mathbf{x}_i + \mathbf{t})\right\|^2, \end{eqnarray} where $P\in\mathbf{R}^{2\times 3}$ is a $2\times 3$ projection matrix that can be assumed to be of rank 2.
let $x_1=(1,0,1), x_2=(-1,0,-1), y_1=(1,0), y_2=(-1,0)$ $$P = \pmatrix{1 & 0 & 0 \\ 0 & 1& 0}$$ $$R_1=\pmatrix{1 & 0 & 0 \\ 0 & 1& 0 \\ 0 & 0& 1}$$ $$R_1=\pmatrix{0 & 0 & 1 \\ 0 & 1& 0 \\ -1 & 0& 0}$$
$$\forall i,j \in \{1,2\}, PR_ix_j=y_j$$
Therefore even the unicity of $R$ is not guaranteed