I would like to know how to find the derivative of $f(\mathbf{x})$ w.r.t. $\mathbf{x}$ where $\mathbf{x} \in \mathcal{V}$ and $\mathcal{V} \subseteq \mathbb{R}^n$.
In particular, what is the derivative of: $$f(\mathbf{x})=\frac{\mathbf{x}^{\top}\mathbf{A}\mathbf{x}}{\lVert\mathbf{x}\rVert^2}$$ where $\mathbf{x} \in \mathcal{V}^{\perp}$ [Edited: as pointed out H.H. Rugh] and $\mathcal{V}$ is the eigenspace corresponding to the top $d(<n)$ eigenvalues of $\mathbf{A}$?
I heard this is called the Gateaux derivative. Any suggestion in this regard will be highly appreciated.
When taking the derivative of $g(x)=x^T B x$ (B symmetric) in the direction of $h$ you calculate as follows: $$ g(x+h) = x^T B x + h^T B x + x^T B h + o(h)= g(x) + g'(x).h + o(h),$$ whence $g'(x).h = 2 h^T B x $. I suspect in your case $A$ is symmetric (or else replace it by $1/2(A+A^T)$. Apply the previous to that ratio and after reduction you should arrive at: $$ f'(x).h = 2\frac{h^T ( A x - f(x) x)}{\|x\|^2}, $$ from which conclusion about critical points may be reached. To look at the set-complement of $V$ is a bit funny. I think it would be more interesting to look at the orthogonal complement (minus the origin).