My problem is: we've got tracking device and a robot. Tracking device provides set of $n$ points in cartesian coordinates(taken from marker on robot arm) and robot driver returns position of TCP(tool center point) also in cartesian coordinates, but in it's own coord system. Given these two sets we would like to calculate how one coordinate origin point relates to another(translation TX, TY, TZ, rotation RX, RY, RZ). I know it's possible, because I saw application do that, but: 1. it's closed source proprietary, 2. I can't use it, 3. I only saw demonstration of how it works, but couldn't ask more questions about internals.
Now, I tried linear equations systems, but I got way more points than needed, so I assume we should use Least Squares method, but I don't have any idea how to implement objective function to do this. After that I think I can come up with some approximate value guesses to start iterations and solve it. I also tried S. Umeyama method described in paper from 1991, but it doesn't yield proper results.
Or is it nonlinear and Gauss-Newton method would be applicable?
Please ask if you have any questions, as English is not my native language and sometimes I struggle to state things clearly, especially when talking about mathematics.
//edited: as I made critical mistake in problem wording
Since your post, a 2D version of the problem has appeared: Transformation from one basis to another
Call the starting vector $Q$, and the transported version $\tilde{Q}$. Define $\theta$ as the rotation angle in the orthogonal bases $x_{k}$, $k=1,2,3.$ The rotation matrices are $$ \mathbf{R}_{1}\left(\theta_{1}\right) = \left[ \begin{array}{crc} \cos \theta & -\sin \theta & 0 \\ \sin \theta & \cos \theta & 0 \\ 0 & 0 & 1 \end{array} \right] \\[4pt] % \mathbf{R}_{2}\left(\theta\right) = \left[ \begin{array}{ccc} \cos \theta & 0 & -\sin \theta \\ 0 & 1 & 0 \\ \sin \theta & 0 & \cos \theta \end{array} \right] \\[4pt] % \mathbf{R}_{3}\left(\theta\right) = \left[ \begin{array}{ccc} 1 & 0 & 0 \\ 0 & \cos \theta & -\sin \theta \\ 0 & \sin \theta & \cos \theta \end{array} \right] $$
The inputs are the translation vector $P$, and the three rotation angles $\theta_{k}$, $k=1,2,3$. Then a vector $Q$ can be moved into $\tilde{Q}$, in the new coordinate system via: $$ \tilde{Q} = P + \mathbf{R}_{1}\left( \theta_{1} \right) \mathbf{R}_{2}\left( \theta_{2} \right) \mathbf{R}_{3}\left( \theta_{3} \right) Q $$