Given matrix $A \in \mathbb{R}^{k \times n}$, define scalar field $F : \mathbb{R}^{n}\ \backslash \{ 0\} \to \mathbb{R}$ by $$F(x) := \frac{|Ax|^2}{|x|^2}$$ and find $\nabla F$
I have tried to write it out by the definition of a partial derivative but end up with a mess of an equation.
On
$
\def\a{\alpha}\def\b{\beta}\def\g{\gamma}\def\t{\theta}
\def\l{\lambda}\def\s{\sigma}\def\e{\varepsilon}
\def\n{\nabla}\def\o{{\tt1}}\def\p{\partial}
\def\E{{\cal E}}\def\F{{\cal F}}\def\G{{\cal G}}
\def\L{\left}\def\R{\right}
\def\LR#1{\L(#1\R)}
\def\bR#1{\big(#1\big)}
\def\BR#1{\Big(#1\Big)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
$Define the scalar functions (and their differentials)
$$\eqalign{
\b &= \|Ax\|^2 \;&=\; x^TBx &\qiq d\b=2Bx:dx,\quad &B=A^TA \\
\g &= \|x\|^2 \;&=\; x^TIx &\qiq d\g=2Ix:dx,\quad &I={\rm Identity} \\
}$$
Use these to rewrite your own scalar function, then calculate its differential and gradient.
$$\eqalign{
\l &= \g^{-1}\b \\
d\l &= \g^{-1}d\b \;-\; \b\g^{-2}d\g \\
&= \g^{-1}\bR{2Bx:dx} - \b\g^{-2}\bR{2Ix:dx} \\
&= 2\g^{-1}\bR{Bx-\g^{-1}\b x}:dx \\
\grad{\l}{x}
&= 2\g^{-1}\bR{Bx-\l x} \\
}$$
which can be rewritten in terms of the original variables
$$\eqalign{
\grad{F}{x}
&= \frac{2}{\|x\|^2} \LR{A^TAx-\LR{\frac{\|Ax\|^2}{\|x\|^2}}x} \\
}$$
You can view that as compositions/operations of functions:
Notice first that since for any $h\in\Bbb R^n$, $A(x+h) = Ax + Ah = Ax + Ah + o(h)$, the differential of $A:x\mapsto Ax$ is given by $\mathrm dA(x) = A$ and so $$ ∇A= A^T $$ In particular, $∇ x= I_n$ is the identity. Then, since $|x|^2 = x· x$, by the product rule $∇(|x|^2) = 2\,x$. By the rule for taking differentials of composition of functions you get $\nabla(|Ax|^2)=2\,A^TA\,x$. Now we are ready to compute the result. $$ \nabla\left(\frac{|Ax|^2}{|x|^2}\right) = \frac{2\,|x|^2\,A^TA\,x-2\,|Ax|^2\,x}{|x|^4} = \frac{2}{|x|^2} \,A^TA\,x - \frac{2\,|Ax|^2}{|x|^4}\,x $$