I recently came across the following problem that resembles a Procrustes problem and I wonder if an analytic solution for this problem might exist:
$$\underset{(R,\alpha)}{\operatorname{argmin}} ||RAe^{j\alpha W}-B||_F$$
Where $A,B \in \mathbb{C}^{3 \times 3}$, $R\in SO_3$, $\alpha\in \mathbb{R}$ and $W=\text{diag}(k_1 w_0,k_2 w_0,k_3 w_0)\in\mathbb{R}^{3 \times 3}$ with $k_1,k_2,k_3\in\mathbb{N}$.
The problem has a similar form to the Orthogonal Procrustes Problem, however there is an additional factor $e^{j\alpha W}$ where $W$ is a real diagonal matrix and $\alpha$ is a variable to be optimized alongside $R$.
My initial solution approach to this looks like this and follows the Wikipedia article:
Similar to the Wikipedia article, I concluded that the solution to the optimization problem should be equivalent to the following with $A'=Ae^{j\alpha W}$:
$$ \underset{(R,\alpha)}{\operatorname{argmax}} \text{Re}( \left \langle R, BA'^{H} \right \rangle_F)$$
$$ \underset{(R,\alpha)}{\operatorname{argmax}} \text{Re}(\left \langle R, Be^{-j\alpha W}A^{H} \right \rangle_F)$$
I thought I might be able to combine $R$ and $e^{-j\alpha W}$ at some point to a unitary matrix using the cyclic property of the Frobenius inner product and then derive an optimal value for this unitary matrix, however, this turned out not to be possible.
Does anyone have another idea on how I could approach this problem?
Additional context about this problem
I am adding some additional context about this problem if that might help: The matrix $A$ can be considered to be a noisy measurement $A=\tilde{A}+\mathcal{N}_A$ where $\mathcal{N}_A \in \mathbb{C}^{3\times 3}$ is a complex noise matrix. The "noiseless" matrix $\tilde{A}$ is related to $B$ via
$$ \tilde{R}\tilde{A}e^{j\tilde{\alpha}}=B$$
I also added some additional information about the matrix $W$.
Update In the book Procrustes Problems I found an iterative algorithm (algorithm 8.2 in section 8.3.6) for finding the optimal weights of a weighted orthogonal Procrustes problem of the form:
$$\underset{(R,C)}{\operatorname{argmin}} ||RAC-B||_F$$
The algorithm solves the above problem for $R\in SO_3$ and $C=\text{diag}(c_1,c_2,c_3)\in \mathbb{R}^3$. I am currently trying to extend this algorithm to complex matrices and apply it to my problem.
Notation.
By definition the Frobenius norm of two matrices is $\langle C,D\rangle_F=\sum_{j,k} c_{j,k} \overline{d_{j,k}}=tr( CD^*)$ where $D^*$ is the conjugate transpose. This is equivalent to $\langle C,D\rangle_F=\sum_j C_j\cdot \overline{D_j}$ where the subscript $j$ indexes the vectors that are the respective rows of $C$ and $D$. (Equivalently we can sum over columns, which is easier to typeset and will be done below.) We will write matrices as lists of column vectors. Thus e.g. $A= [A_1, A_2, A_3]$ and $B=[B_1, B_2, B_3]$.
The constant $w_0$ in $W= diag( k_1 w_0, k_2 w_0, k_3 w_0)$ can be absorbed within the variable $\alpha$ that defines $ e^{i\alpha W}$. Denote the latter diagonal matrix by $ e^{i \alpha W}=U= diag ( u_1, u_2,u_3)$ where each $u_j = e^{i k_j \theta}$ is a complex number of length $ 1$.
Re-statement of problem
As you noted in your posting, the goal is to maximize $\Re \langle RA U, B \rangle =\Re \langle AU, R^tB \rangle$
In this column format,
$AU=[u_1 A_1, u_2 A_2, u_3 A_3]$ and
$M=R^tB$ is a linear combination of columns of $B$, of the form $[M_1, M_2, M_3]$ where $M_j= \sum_k r_{j,k} B_k$.
Thus $\Re \langle AU, M\rangle = \sum_{j,k} r_{j,k} u_j \langle A_j , B_k\rangle$
Define $C_{j,k}= \langle A_j, B_k\rangle$, which is a collection of Hermitian inner products of columns of $A$ and $B$. Note that $\Re \langle AU, M\rangle= \Re \langle RU, C\rangle_F$.
The goal is to select $R$ and $U$ so that the left-hand matrix $RU$ approximates a given constant matrix $C$ as closely as possible in the F-norm. Note that $RU$ is a unitary matrix.
From this point I suggest exploring the properties of the singular value decomposition of the complex matrix $C$