Consider $f(x) = \frac{1}{\|Ax\|_2^2}$, where $A \in \mathbb{R}^{m \times n}$ I want to compute the hessian of $f$. For the gradient I end up with: $$ \nabla f(x) = \frac{-2 A^TAx}{\|Ax\|_2^4}$$. Now for the hessian I tried the following:
$$ \nabla^2 f(x) = \frac{-2 A^TA + 8 \|Ax\|_2^3 2 ( A^T A x)^T A^T A x }{\|Ax\|_2^8}$$
I think I did a mistake here. Can maybe somebody clarify what I have done wrong?
Assuming the gradient calculation is correct, start with:
$\begin{equation} \nabla^2f(x) = \frac{(||Ax||^4)(-2A^TA)+(2A^TAx)(4||Ax||^3)}{||Ax||^8} \end{equation}$
This is just the quotient rule (low dhigh - high dlow all over low squared). Now, we can make some simplifications:
$\begin{align} \nabla^2f(x) &= \frac{-2A^TA||Ax||^4 + 8A^TAx||Ax||^3}{||Ax||^8}\\ &= \frac{-2A^TA||Ax|| + 8A^TAx}{||Ax||^5}\\ &= \frac{-2A^TA(||Ax||-4x)}{||Ax||^5} \end{align}$
And of course this is all with respect to the $\ell^2$ norm but it should also work in general. Let me know if this makes sense or if I made any mistakes. Hope it helps!