I am working with 3 matrices, $A$,$B$, and $R$, where $A$ and $B$ are of dimension $3 \times N$ (N is the number of datapoints) and $R$ is a $3 \times 3$ matrix with a very specific setup: $$ R = \begin{bmatrix} \cos(\theta) & -\sin(\theta) & 0 \\ -\sin(\theta)& -\cos(\theta) & 0\\ 0& 0 & -1 \end{bmatrix} $$
Now I am trying to minimize the following: $$ \min_{R}||A - RB||^2 $$ Where R is subject to being of that form up top.
Now my question is simple, what's the best way for me to achieve this minimization while in turn satisfying the specific setup that R has?
Suggestion:
I am assuming Frobenius norm. If so, then what really matters are the first two rows of $A$ and $B$ and you only care about $$ R = \begin{bmatrix} \cos(\theta) & -\sin(\theta)\\ -\sin(\theta)& -\cos(\theta)\\ \end{bmatrix} $$ You can write $R$ in a vector form $(a, b, c ,d)$, where you add three constraints $a^2+b^2 =1$, $b = c$, and $ a = -d$. This finally gives a quadratically constrained quadratic program.