What is the Hessian matrix of the square root of a quadratic form: $\left(w^T H w\right)^{0.5}$?
Got the gradient, $0.5 \left(w^T H w\right)^{-0.5} ( 2 H w)$, which gives numerically correct results, but I fail with classical differentiation rules in calculating the Hessian.
Let's focus on partial derivatives. For $f:\mathbb R^n\to \mathbb R$ and $g:\mathbb R \to\mathbb R$, we can differentiate $g\circ f$ using the chain rule: $$\frac{\partial (g\circ f)}{\partial w_i} = g'(f) \frac{\partial f}{\partial w_i}$$ This agrees with your computation, where $2Hw$ is the gradient of $f$ and $\frac12 f^{-1/2}$ is $g'(f)$.
Let's do this again, with respect to $w_j$: $$\frac{\partial^2 (g\circ f)}{\partial w_i \partial w_j} = (g'(f))^2 \frac{\partial^2 f}{\partial w_i \partial w_j} + g''(f) \frac{\partial f}{\partial w_i} \frac{\partial f}{\partial w_j}$$ With your $g$ and $f$, this is $$ \frac{1}{2(w^T Hw)} H_{ij} - \frac{1}{(w^T Hw)^{3/2}} (Hw)_i (Hw)_j $$ where the indices select the entries of matrix or vectors. In matrix form, you can write this using the outer product as $$ \frac{1}{2(w^T Hw)} H - \frac{1}{(w^T Hw)^{3/2}} Hw \otimes Hw $$