Find the value of unit vector that maximizes the following cost function

Question

Let us consider the following cost function:

$$J(\mathbf{A},\mathbf{B}) = \frac{\mathbf{u}^T\mathbf{Au}}{\mathbf{u}^T\mathbf{Bu}}$$

where, $\mathbf{A}$ and $\mathbf{B}$ are given symmetric positive definite matrices and variable $\mathbf{u}$ is a unit vector. I want to know the steps for finding the value of $\mathbf{u}$ that maximizes $J$. Additionally, I have the following queries:

1. Is maximizing $J$ equivalent to maximizing the numerator?
2. Is maximizing $J$ equivalent to minimizing denominator?
3. Is it possible to get two different answers for points 1 and 2?

Answer 1

Define a few scalar variables and their differentials $$\eqalign{ \alpha &= A:uu^T &\implies d\alpha = 2Au:du \cr \beta &= B:uu^T &\implies d\beta = 2Bu:du \cr }$$ where a colon denotes the trace/Frobenius product, i.e. $\,\,\,A\!:\!B={\rm tr\,}(A^TB)$

Write the cost function in terms of these, then find its differential and gradient $$\eqalign{ J &= \frac{\alpha}{\beta} \cr dJ &= \beta^{-2}({\beta\,d\alpha-\alpha\,d\beta}) \cr &= 2\beta^{-2}({\beta Au-\alpha Bu}): du \cr &= 2\beta^{-1}(Au-JBu) : du \cr \frac{\partial J}{\partial u} &= 2\beta^{-1}(Au-JBu) \cr }$$ Set the gradient to zero and solve $$\eqalign{ Au &= JBu \cr (B^{-1}A)u &= Ju \cr }$$ This is just an eigenvalue equation. So $J$ is the largest eigenvalue of $\,(B^{-1}A)$ and $u$ is the associated eigenvector.

Answer 2

No, it is not as simple as maximizing the numerator or minimizing the denominator separately. These will give different answers in general. $u^\top Au$ is maximized when $u$ is any eigenvector with maximal eigenvalue, while $u^\top Bu$ is minimized when $u$ is a minimal eigenvector. There is no reason these should be related, because $A$ and $B$ are unrelated.

Your program is $$ \begin{aligned} &\text{maximize}_u & &\frac{u^\top Au}{u^\top Bu}\\ &\text{subject to} & &u^Tu=1 \end{aligned} $$ The constraint $u^Tu=1$ is unnecessary, since scaling $u$ does not affect $J(u)$. So we may drop this constraint, and instead impose the constraint $u^\top Bu=1$. This simplifies the problem since the denominator is now $1$, and we have the equivalent program $$ \begin{aligned} &\text{maximize}_u & &{u^\top Au}\\ &\text{subject to} & &{u^\top Bu}=1 \end{aligned} $$ This can be solved via Lagrange multipliers, for example.

Answer 3

Since $\rm B$ is positive definite, it has an invertible square root. Let $\mathrm v := \mathrm B^{\frac 12} \mathrm u$. Hence,

$$\frac{\mathrm u^\top \mathrm A \,\mathrm u}{\mathrm u^\top \mathrm B \,\mathrm u} = \frac{\mathrm v^\top \mathrm B^{-\frac 12} \mathrm A \mathrm B^{-\frac 12} \mathrm v}{\mathrm v^\top \mathrm v} \leq \lambda_{\max} \left( \mathrm B^{-\frac 12} \mathrm A \mathrm B^{-\frac 12}\right) = \lambda_{\max} \left( \,\, \mathrm A \mathrm B^{-1} \right)$$