Computing the derivative of $\|Ax\|_2$

Question

Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $A$ is an arbitrary matrix and $x$ is a vector.

I think somebody said that the result is $2A^TAx$, but I cannot get even there. I have no idea how to develop this norm and derive it in matrix form. Because if it were $$\frac{\partial \|x\|}{\partial x}=\frac{\partial x^Tx}{\partial x}=x$$ but with that matrix in the middle. I do not know how to solve that. Thanks in advance.

Answer 1

In the Euclidean norm ($p$ norm for $p=2$) $$ \lVert y \rVert = \left( \sum_i y_i^2 \right)^{1/2} $$ for the $k$–th coordinate of the gradient by applying the chain rule several times we have $$ \begin{align} \partial_k \lVert Ax \rVert &= \frac{\partial}{\partial x_k} \left( \sum_i \left( \sum_j a_{ij}x_j \right)^2 \right)^{1/2} \\ &= \frac{1}{2 \lVert Ax\rVert} \sum_i 2 \left( \sum_j a_{ij}x_j \right) a_{ij} \delta_{jk} \\ &= \frac{1}{\lVert Ax\rVert} \sum_i a_{ik} \left( \sum_j a_{ij}x_j \right) \\ &= \frac{1}{\lVert Ax\rVert}\left( A^T A x \right)_k \end{align} $$ Note:Funny enough here I was able to recycle another answer from this morning.

Answer 2

It is useful to introduce the Frobenius inner product as:

$$ A:B = \operatorname{tr}(A^TB)$$

with the following properties derivied from the underlying trace function

$$\eqalign{A:BC &= B^TA:C\cr &= AC^T:B\cr &= A^T:(BC)^T\cr &= BC:A \cr } $$

Then we work with differentials to find the gradient. Your problem becomes, with $u=Ax$

$$\eqalign{ f &= \|u\|_{F}^{2} = u:u \\ df &= 2u : du\\ &= 2Ax : A dx\\ &= 2A^TAx : dx} $$

Thus

$$ \frac{\partial f}{\partial x} = 2 A^TAx$$

EDIT:

For $g = \|u\|_{2} $ this becomes:

$$\eqalign{ g&= f^{1/2} \\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : df\\ dg &= \frac{1}{2} f^{-\frac{1}{2}} : 2u : du\\ &=f^{-\frac{1}{2}} u : du\\ &=\frac{1}{||Ax||} Ax : Adx\\ &=\frac{1}{||Ax||} A^TAx : dx\\ }$$

$$\frac{\partial g}{\partial x}= \frac{1}{||Ax||}A^T A x$$