I have the following system of equations
$$X R a - c_1 = 0$$ $$X a - c_2 = 0$$ $$X X^T=Identity$$
where $X\in\mathbb{R}^{3x3}$, $a,c_1,c_2\in\mathbb{R}^{3x1}$, $R\in\mathbb{R}^{3,3}$ is a rotation matrix. For some $a,R,c_1,c_2$ the system has two possible $X$'s as a solution. In cases where there is no solution I want to find an $X$ which minimizes the "error" while maintaining that $XX^T=Identity$. How can this be done?
Would knowing the following help? $$a = (0,0,-1)$$ $$|c_1|=1$$ $$|c_2|=1$$ $$X \text{ is a rotation matrix}$$
The condition $XX^T = I$ means $X$ is orthogonal. So if the system has no solution, you would like to project an approximate solution (where $X$ is nearly orthogonal) onto its "orthogonal part", which is the "Orthogonal Procrustes problem". The solution method is described there. Summarizing, let $(Ra|a)$ be the $3 \times 2$ matrix whose first colument is $Ra$ and whose second column is $a$ and similarly for $(c_1|c_2)$. If we were to write $X(Ra|a) - (c_1|c_2) = 0_{3 \times 2}$, we would exactly capture your first two constraints. Instead, we will minimize this expression relative to the Frobenius norm, which here is equivalent to the Euclidean matrix norm (the square root of the sum of the square of each of the six components). \begin{align*} \text{minimize } &||X \left( Ra \middle| a \right) - \left( c_1 \middle| c_2 \right) ||_F \\ \text{subject to } &XX^T = I \end{align*}
Compute the singular value decomposition of $(c_1 | c_2)(Ra|a)^T$. $$ (c_1|c_2)(Ra|a)^T = U \Sigma V^T $$ then the projection of the solution onto the orthogonal component is $$ X = UV^T \text{.} $$
(I've never actually implemented this algorithm, so a typo' or missing "T"ranspose in the above might slip past me. If you find such a thing, I'm happy to fix it.)