In the introduction part of the paper The Fast Convergence of Incremental PCA, the authors mention that the gradient of the Rayleigh quotient is equal to:
$$ \triangledown G(v) = \frac{2}{\|v\|^2}(A - \frac{v^{T}Av}{v^{T}v} I_d)v $$
when the Rayleigh quotient is: $$G(v) = \frac{v^{T}Av}{v^{T}v}$$
($v \in \mathbb{R}^d$ and $A \in \mathbb{R}^{d\times d}$)
What are the steps to derive the given value for $\triangledown G(v)$ ?
EDIT:
As suggested by @Alex R. in a comment, I tried to proceed using the identity for a derivative of a quotient. I don't know/remember the matrix calculus identities to proceed. Here's what I tried:
Let $N = v^{T}Av$ and $D = v^{T}v$,
Then, $G'(v) = \frac{N'D-ND'}{D^{2}} \tag{1}\label{eq1}$
$D^2$ can be written as $(v^{T}v)^2 = (\|v\|^{2})^2$.
Using $\frac{dx^{T}Ax}{dx} = x^{T}(A + A^{T})$, $N'$ can be written as $v^{T}(A+A^{T})$.
$D' = 2v^{T}$.
Plugging these values to $\eqref{eq1}$ yields:
$$ G'(v) = \frac{v^{T}(A+A^{T})v^{T}v - v^{T}Av(2v^{T})}{\|v\|^4} $$
I can't figure out how to simplfy this. Any pointers to resources I should look are also appreciated.
A covariance matrix is symmetric. Hence $A^T = A$. Thus
$$G'(v) = \frac{v^{T}(A+A^{T})v^{T}v - v^{T}Av(2v^{T})}{\|v\|^{\displaystyle \color{red}{4}}} =\frac{\color{green}{v^{T}}\color{blue}{(2A)}v^{T}v - (2\color{green}{v^{T} I_d})v^{T}Av}{\|v\|^{4}} = 2\color{green}{v^T}\frac{A\|v\|^2 - I_dv^T Av}{\|v\|^4}= \frac{2v^T}{\|v\|^2}\left(A - \frac{v^TAv}{\|v\|^2}I_d\right) $$
which is exactly the result, after you realise that you and the author of the paper have used different conventions for gradient, see e.g. here.