How to find gradient and hessian of summation

Question

I am confused because of the summation and its implicit form. Could you please help me to find gradient and Hessian of the function given below?

Thanks a lot.

$$ f(x) = \Vert x\Vert = \left(\sum_{i=1}^nx_i^2\right)^\frac 12 $$

Answer 1

You are sort of close. But you treat all the $x_i$ as if they were $x$. Also, the answer should be an $n$-dimensional vector.

The gradient is $\nabla f = (\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}, \ldots, \frac{\partial f}{\partial x_n})$, so we need to work out $\frac{\partial f}{\partial x_i}$, i.e. the partial derivatives. Using the chain rule: $$ \frac{\partial f}{\partial x_i} = \frac{\partial}{\partial x_i} \left(\sum_{i=1}^n x_i^2\right)^\frac12 = \frac12 \left(\sum_{i=1}^n x_i^2\right)^{-\frac12} \cdot \left(\frac{\partial}{\partial x_i} \sum_{i=1}^n x_i^2\right) = \frac12 \left(\sum_{i=1}^n x_i^2\right)^{-\frac12}\cdot 2x_i = \frac{x_i}{\Vert x\Vert} $$ (Do you see why the third equality is right?) Therefore, the gradient is $$ \nabla f = \left(\frac{x_1}{\Vert x\Vert},\ldots,\frac{x_n}{\Vert x\Vert}\right) = \frac{x}{\Vert x\Vert} $$ There are other ways to show the result (probably the most elegant using $\Vert x\Vert^2 = x\cdot x$ with the product rule), but this was the most basic way.

Hopefully you can go on and find the Hessian now?

[Hessian hint]: For the Hessian, you need to work out all possible double derivatives: $$ \frac{\partial^2f}{\partial x_j\partial x_i} = \frac{\partial}{\partial x_j} \frac{\partial f}{\partial x_i} = \frac{\partial}{\partial x_j} \frac {x_i}{\Vert x\Vert} $$ The result will depend on wether $j=i$ or $j\ne i$. In the case $j=i$, you need to use the quotient rule. Also, remember that we already know what $\frac{\partial}{\partial x_j}\Vert x\Vert$ is, which you will need.

If this is too complicated, try the simple case of two dimensions first: $\Vert (x,y)\Vert = (x^2+y^2)^\frac 12$, and find the Hessian for this function, and then try to generalise.

Answer 2

Once you have convinced yourself that $$\frac{\partial f(x)}{\partial x_i}=\frac{\partial \|x\|}{\partial x_i}=\frac{x_i}{\|x\|}$$ Then, recall that, the Hessian matrix of $f$ is the $n\times n$ matrix $\textbf H$ with the $(i,j)$-entry given by $$\textbf{H}_{ij}=\frac{\partial^2 f(x)}{\partial x_j\partial x_i}=\frac{\partial}{\partial x_j}\bigg(\frac{\partial f(x)}{\partial x_i}\bigg)$$ thus, we will have to calculate the latter in order to give the general input of the matrix.

Using the quotient rule, we see that $$\begin{align} \frac{\partial}{\partial x_j}\bigg(\frac{\partial f(x)}{\partial x_i}\bigg) &= \frac{\partial}{\partial x_j}\bigg(\frac{x_i}{\|x\|}\bigg)=\cfrac{\cfrac{\partial x_i}{\partial x_j}\cdot\|x\|-x_i\cdot\cfrac{\partial \|x\|}{\partial x_j}}{\|x\|^2} \\ &= \frac{1}{\|x\|}\frac{\partial x_i}{\partial x_j}-\frac{x_ix_j}{\|x\|^3} \end{align}$$ Note that, $$\frac{\partial x_i}{\partial x_j} = \delta_{ij} =\begin{cases} 1 & \textrm{if } i=j \\ 0 & \textrm{if } i\neq j \end{cases}$$ Hence, we have $$\mathbf{H}_{ij}=\frac{1}{\|x\|}\delta_{ij}-\frac{x_ix_j}{\|x\|^3} = \frac{\delta_{ij}\|x\|^2-x_ix_j}{\|x\|^3}$$ as, for example, if $n=2$, then $f$ is given by $f(x,y)=\|(x,y)\|=\sqrt{x^2+y^2}$ and then $$\mathbf{H}=\begin{pmatrix} \cfrac{\delta_{11}\|(x,y)\|^2-x^2}{\|(x,y)\|^3} & \cfrac{\delta_{12}\|(x,y)\|^2-xy}{\|(x,y)\|^3} \\ \cfrac{\delta_{21}\|(x,y)\|^2-yx}{\|(x,y)\|^3} & \cfrac{\delta_{22}\|(x,y)\|^2-y^2}{\|(x,y)\|^3} \end{pmatrix}=\begin{pmatrix} \cfrac{y^2}{(x^2+y^2)^{3/2}} & -\cfrac{xy}{(x^2+y^2)^{3/2}} \\ -\cfrac{xy}{(x^2+y^2)^{3/2}} & \cfrac{x^2}{(x^2+y^2)^{3/2}} \end{pmatrix} = \frac{1}{(x^2+y^2)^{3/2}}\begin{pmatrix} y^2&-xy \\ -xy&x^2 \end{pmatrix}$$