Let $\mathbf{A}$ and $\mathbf{B}$ be real matrices with $\mathbf{A}\overset{\mathrm{SVD}}{=}\mathbf{U}_A \mathbf{S}_A \mathbf{V}_A^\mathrm{T}$. I want to maximize \begin{align} \max_B \mathrm{trace}(\mathbf{A}\mathbf{B}) \quad \mathrm{with} \quad \mathbf{B}^\mathrm{T}=\mathbf{B}^{-1} \; \wedge \; \mathrm{det}(\mathbf{B})=1, \end{align} i.e. $\mathbf{B}$ is a proper rotation matrix while $\mathbf{A}$ is some fixed general matrix.
Without the constraint, we know from von Neumann's trace inequality that the absolute value of the term is maximized if both matrices share the same set of singular values. Only considering the first constraint, this would lead to $\mathbf{B}=\mathbf{V}_A \mathbf{U}_A^\mathrm{T}$ and works well as long as $\mathrm{det}(\mathbf{U}_A\mathbf{V}_A)=1$. That is, it really maximizes the $\mathrm{trace}$ and not its absolute value.
Own considerations: However, I think that in case that $\mathrm{det}(\mathbf{U}_A\mathbf{V}_A)=-1$ this would require $\mathbf{B}$ to be a reflection as well and thereby violate the second constraint. Considering this second requirement, I believe the prior solution is not feasible anymore and the new maximization would be obtained by choosing $\mathbf{B}=\mathbf{V}_A \mathrm{diag}(1,1,-1) \mathbf{U}_A^\mathrm{T}$ if the singular values of $\mathbf{S}_A$ are ordered. This solution would negate the smallest eigenvalue of the term and thereby slightly reduce the result. However, this is just my gut feeling and I have no prove for this.
I would highly appreciate if someone could help me solve this issue or contradict/verify my own considerations!
Edit: If there is anything I can do to make this problem more clear or to illustrate why I believe in the latter solution of $\mathbf{B}$ please let me know.