I am facing an optimization problem where I want to maximize $\operatorname{trace} (U V)$ with respect to $U$ and $V$ where $U$ and $V$ are orthogonal. How can I find the optimal $U$ and $V$?
This is part of a longer problem where $U$ and $V$ are left singular vectors of unknown matrices.
The full problem is:
$$\max_{U,V} \quad \operatorname{trace}(UV) $$
s.t.
$D_1 = U\Sigma_1 V_1$
$D_2 = V\Sigma_2 V_2$
$D_1 = D_1'x_1x_2$
$D_2 = D_2'y_1y_2$
$D_1' \quad and \quad D_2'$ are known. I am looking for $x_1,x_2,y_1, y_2$ so that the trace is maximized. Any help or hint about where to start e.g. a paper, a course, etc. is very appreciated.
Since, if $V$ is orthogonal, then it is also $V^{T}$, the problem of your title is equivalent to
$$\max_{U, V} \text{tr}(U V^{T}) \qquad \text{s.t} \qquad U \text{ and } V \text{ are orthogonal.}$$
Now, since $\text{tr}(U V^{T})$ is an inner product from which the Frobenius norm ($\|\cdot \|_{F}$) is inherited, we have that $\text{tr}(U V^{T}) \leq \|U\|^2_{F} \|V\|^2_{F}$. Observe that $\|U\|^2_{F} \|V\|^2_{F} = n^2$ for any pair of orthogonal matrices. Now, it is easy to see that the maximum is achieved at $U=I$ and $V=I$.
P.S.: Regarding your full problem, I can't even see why $U$ and $V$ are orthogonal. If $U$ and $V$ are left singular vectors of unknown matrices, this needs to be stated at the constraints somehow. For me, your full problem is ill-formulated.