The generalized Rayleigh ratio is defined by
$$R(\vec{x})=\frac{\vec{x}^TA\vec{x}}{\vec{x}^TB\vec{x}}.$$
The vector $\vec{x}$ is a $p \times1$ unitvector, the matricies $A$ and $B$ have the size $p \times p$. Additionally $A$ is positive semi definit and $B$ is positive definit.
How can I maximize this expression? I tried to rewrite the lower term as a inverse matrix and then use the external definition of the gradient, but that did not work out.
Let $$\eqalign{ \alpha &= x^TAx \cr \beta &= x^TBx \cr R &= \lambda = \frac{\alpha}{\beta} \cr }$$ Then take the gradient of $\lambda$ (with respect to $x$) and set it to zero $$\eqalign{ \nabla\lambda & =\frac{\beta\,\nabla\alpha-\alpha\,\nabla\beta}{\beta^2} = 0 \cr \beta\,\nabla\alpha &= \alpha\,\nabla\beta \cr \nabla\alpha &= \lambda\,\nabla\beta \cr 2Ax &= 2\lambda\,Bx \cr B^{-1}Ax &= \lambda\,x }$$ So the maximum value of $R$ is the maximum eigenvalue of $B^{-1}A$.