Maximizing $\frac{u^T P u}{u^T S u}$ w.r.t. $u$ of unit length

51 Views Asked by At

Is there a closed form (or efficient) solution to $$\max_{\Vert u \Vert_2 = 1} \frac{u^T P u}{u^T S u}$$ where $u\in\mathbb{R}^k$, $S\in\mathbb{R}^{k\times k}$ is PD, and $P\in\mathbb{R}^{k\times k}$ is rank 1 PSD?

If anyone's curious, this optimization arises from the following problem: choose a direction $u$ to project a random vector $X$ onto such that the resulting vector is maximally correlated with a random scalar $y$. You will get the above optimization, with $S=\Sigma_X$ and $P=\Sigma_{Xy} \Sigma_{Xy}^T$.

1

There are 1 best solutions below

1
On BEST ANSWER

In general, when $S$ is PD and $P$ is symmetric, $$ \max_{\|u\|_2=1}\frac{u^TPu}{u^TSu} =\max_{u\ne0}\frac{u^TPu}{u^TSu} =\max_{v=S^{1/2}u\ne0}\frac{v^TS^{-1/2}PS^{-1/2}v}{v^Tv} =\lambda_\max(S^{-1/2}PS^{-1/2}). $$ So, $\max_{\|u\|_2=1}\frac{u^TPu}{u^TSu}$ is maximised at $u=\frac{S^{-1/2}v}{\|S^{-1/2}v\|_2}$, where $v$ is any eigenvector corresponding to the maximum eigenvalue of $S^{-1/2}PS^{-1/2}$.

In particular, when $P=ww^T$ is a real rank-one PSD matrix, we have $\lambda_\max(S^{-1/2}PS^{-1/2})=\operatorname{tr}(PS^{-1})=w^TS^{-1}w$, for which $v=S^{-1/2}w$ is an eigenvector. In fact, the Rayleigh quotient $\frac{v^TS^{-1/2}PS^{-1/2}v}{v^Tv}$ is equal to $w^TS^{-1/2}\frac{v}{\|v\|_2}$. One easily sees that it is maximised when $v=S^{-1/2}w$. At anyway, $u=\frac{S^{-1/2}v}{\|S^{-1/2}v\|_2}=\frac{S^{-1}w}{\|S^{-1}w\|_2}$ is a global maximiser in this case.