Maximize trace of matrix equation given two constraints

Question

Let $\mathbf{Q}$ be a rotation matrix and $\mathbf{A}$ and $\mathbf{B}$ be two real-valued matrices of the same size. I want to maximize the function $$ f(\mathbf{Q})=tr\;\mathbf{QA} \qquad \text{s.t.} \; \mathbf{Q}'\mathbf{Q} = \mathbf{I} \quad \text{and} \quad tr\; \mathbf{QB} \geq 0 $$

If only the orthogonality constraint is imposed, the solution is $$ \mathbf{Q} = \mathbf{VU}' $$

(see e.g. Cliff, 1966) with $\mathbf{U}$ and $\mathbf{V}$ from the singular value decomposition of $\mathbf{A} = \mathbf{U \Lambda V}'$.

Question: How can I find a solution that also includes the second constraint, i.e. $tr\; \mathbf{QB} \geq 0$, given that it exists?

Footnote: I am not a mathematician, so please be verbose in your answer, so I have a chance to follow :)

Literature

Cliff, N. (1966). Orthogonal rotation to congruence. Psychometrika, 31(1), 33–42. http://doi.org/10.1007/BF02289455

Answer 1

The SVD of $A$ is $A=U\Lambda V^T$. If $\det(A)\not=0$, then the so-called solution $Q_0=VU^T$ is s.t. $\det(Q_0)=signum(\det(A))=\pm 1$ ; then I assume that $Q\in O(n)$.

Let $C=U^TBV,R=V^TQU\in O(n)$. If $tr(Q_0B)=tr(C)\geq 0$, then the solution is $Q_0$.

Assume that $tr(C)<0$. One has $tr(QA)=tr(R\Lambda)$ ; the sup is reached when $tr(QB)=tr(VRU^TB)=tr(RC)=0$ (not exactly; cf. EDIT 1. below).

Then it remains to find $\sup_{R\in O(n)}tr(R\Lambda)$ under the condition that $tr(RC)=0$.

EDIT 1. Assume that the sup of $g(R)=tr(R\Lambda)$ is reached in $R=R_1$ with $tr(R_1C)>0$. Then $g(R_1)$ is a free local sup of $g$.

Case 1. $R_1\in O^+(n)$. Since $O^+(n)$ is a connected set, $g(R_1)=g(I)=tr(\Lambda)$. If $\Lambda=diag(\lambda_1,\cdots,\lambda_k,0,\cdots,0)$ where $\lambda_i>0$, then $R_1=diag(I_k,S)$ where $S\in O^+(n-k)$.

Case 2. $R_1\in O^-(n)$. Then $g(R_1)=g(J)$ where $J=diag(u_1,\cdots,u_n)$ with $u_i=\pm 1$.

Unfortunately, Case 2 may occur. Take, for instance, $\Lambda=diag(1,1,0,0,0),C=diag(-3,-3,-2,-2,-3)$. Then the sup is reached in a sole point $R_1=diag(1,1,-1,-1,-1)$ with $tr(R_1C)=1$.

EDIT 2. Answer to Mark. I agree with the case $tr(C)\geq 0$ and your Point A.

About your point B where necessarily $tr(R_1C)>0$.

Case 1: $R_1\in O^+(n)$. Then necessarily $R_1=diag(I_k,S)$ and you seek $S$ s.t. $tr(R_1C)>0$. If $S$ exists, then $\max(g)=tr(\Lambda)$, else $R_1$ does not exist.

Case 2: $R_1\in O^-(n)$. Then necessarily $(\star)$ $g(R_1)=max_k(g(J_k))$ where $J_k$ is the diagonal matrix s.t. $(J_k)_{i,i}=1$ if $i\not= k$ and $(J_k)_{k,k}=-1$. You seek $R_1$ satisfying $(\star)$ and s.t. $tr(R_1C)>0$...

Finally, you choose "the" best candidate. Note that Point A is not easy to solve.

EDIT 3. Mark, I do not want to write details; in fact, I use the following results (that are not obvious to prove):

1. Let $O\in O^+(n)$. There is a continuous function $f:[0,1]\rightarrow O^+(n)$ s.t. $f(0)=O,f(1)=I$ and s.t., for every $i$, the function $f_{i,i}$ is non-decreasing.

After discussion with a colleague, a correct statement is:

2. Let $O\in O^-(n)$. There is a continuous function $f:[0,1]\rightarrow O^-(n)$ s.t. $f(0)=O$, $f(1)$ has spectrum $\{1,\cdots,1,-1\}$ and s.t., for every $i$, the function $f_{i,i}$ is non-decreasing.

NB. During such a deformation, $tr(R\Lambda)$ is non-decreasing.