I am solving a least squares optimization.
I have a 2D rigid polygon, which can be completely described through a rigid transformation $T = \begin{bmatrix} R_\theta &t \\ 0 &1\end{bmatrix}$
Note: it's a $3 \times3$ matrix expressed in block notation, R is a $2 \times2$ rotation and t a $2 \times1$ translation).
My state is $X = \begin{bmatrix} t_x & t_y &\theta\end{bmatrix}$
I want to formulate a least square problem such that the polygon is moved to a configuration where it can be described by a "smaller" transformation T.
My cost function is the difference between the transformation T describing the rigid body and the identity transformation $I = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$.
A simple way to compute this difference would be to consider a $6 \times1$ error vector where each term is the element-wise scalar difference between elements in the 2 matrices.
$e^T = \begin{bmatrix}\cos\theta - 1 & -\sin\theta - 0 & t_x - 0 & \sin\theta - 0 & \cos\theta - 1 & t_y - 0 \end{bmatrix}$
The vector is $6 \times1$ and not $9 \times1$ simply because the last row is constant in both matrices.
The error function is non linear, so I will have to search for an iterative solution using Gauss-Newton method.
The jacobian $J = \frac{\partial{e(X + \Delta x)}}{\partial{\Delta x}} $ will be a $6\times3$ matrix.
$J = \begin{bmatrix} 0 &0 &\cos\theta \cos\Delta_\theta - \sin\theta \sin\Delta_\theta\\ 0 &0 &-\cos\theta \sin\Delta_\theta - \sin\theta \sin\Delta_\theta\\ 0 &0 &\sin\theta \cos\Delta_\theta + \cos\theta \sin\Delta_\theta\\ 0 &0 &-\sin\theta \sin\Delta_\theta + \cos\theta \cos\Delta_\theta\\ 1 &0 &- \sin\theta t_{\Delta x} - \cos\theta t_{\Delta y}\\ 0 &1 &\cos\theta t_{\Delta x} - \sin\theta t_{\Delta y}\\ \end{bmatrix}$
Evaluating this jacobian at $\Delta x = 0$ and computing the perturbation as $\Delta x = - (J^T J)^{-1} J^T e$ will lead to the desired minimization.
Now assume that my state is something different: the set of 2D vertices of this polygon, represented through their x-y coordinates: $X = \begin{bmatrix}x_0 &y_0 &x_1 &y_1 &\dots\end{bmatrix}$.
I would like to formulate the same least square problem with respect to the new state.
I am thinking about applying the chain rule such that $J = \frac{\partial{e(X_{new} + \Delta x_{new})}}{\partial{\Delta x_{new}}} = \frac{\partial{e(X_{old} + \Delta x_{old})}}{\partial{\Delta x_{old}}} \frac{\partial{\Delta x_{old}}}{\partial{\Delta x_{new}}} $
How can I express analytically $\frac{\partial{\Delta x_{old}}}{\partial{\Delta x_{new}}} $ ?
I know that for a single point $P_{new} = R P_{old} + t$.
If my transformation would be a simple translation with no rotation, the jacobian I'm looking for could be easily obtained as
$P_{new} = P_{old} + t$
$t = P_{new} - P_{old}$
$\frac{\partial{t}}{\partial{P_{new}}} = \begin{bmatrix} 1 && 0 \\ 0 && 1\end{bmatrix}$
When dealing also with rotations, I need to consider more than 1 point in order to invert the relation somehow...