I want to maximize $Tr(W^TP\Sigma PW)$ where $\Sigma$ is a covariance matrix, $P=uu^T$ is an orthogonal projection matrix over a vector $u$, and the maximization is over all $u$ such that $||u||=1$. Does this problem have a closed-form solution?
Thanks!
Let $\Sigma=VV^T$ (for example via Cholesky decomposition) and $P=\tfrac{uu^T}{u^Tu}$. Setting the derivative of
$$Tr(W^TP\Sigma PW)=\langle V^T \tfrac{uu^T}{u^T u} W, V^T \tfrac{uu^T}{u^T u} W\rangle_F = \|V^T \tfrac{uu^T}{u^T u} W\|_F^2$$
equal to zero yields
$$ \frac{VV^T u}{\|V^T u\|_2^2} + \frac{WW^T u}{\|W^T u\|_2^2} \overset{!}{=} 2\frac{u}{\|u\|^2} $$
or equivalently, the fixed-point equation
$$ \frac{1}{2}\bigg(\frac{\|u\|^2_2}{\|V^T u\|_2^2}VV^T u+ \frac{\|u\|_2^2}{\|W^T u\|_2^2} WW^T u\bigg)\overset{!}{=} u $$
Which I would not expect to admit a closed form solution.
One thing one can try is solve the generalized eigenvalue problem, i.e. find $u$ satisfying $A u =\lambda B u$, where $A=-WW^T$, $B=VV^T$. The wikipedia article mentions that in the case when $A$ and $B$ are Hermitian, and $B$ is positive definite, there exists a basis of EV $u_i$ which are $B$-orthogonal. I tried experimenting with it a bit but couldn't get it into any closed form, but maybe you have some more luck.
At the very least, I suspect that the fixed point iteration may offer a simple and efficient way to solve the problem numerically.