Prove that $\nabla\langle Ax,Ax\rangle = 2A^TAx$

Question

Prove that $\nabla\langle Ax,Ax\rangle = 2A^TAx$.

My book uses this property to prove the $2-norm$ of a matrix $A$ is the square root of the spectral radius of $A^TA$. That is $$||A||_2 = \sqrt{r(A^TA)}$$ where $r(A)$ is the spectral radius. How would one go about proving this?

This isn't a homework question. I would just like to know for completeness of the proof. T

Any help and comments is appreciated. Thank you in advance.

Answer 1

Using the indice summation convention for repeated indices and letting $v=\nabla\langle Ax,Ax\rangle$, we have $$v_i=\partial_i(A_{jk}x_kA_{jl}x_l)=A_{jk}\delta_{ik}A_{jl}x_l+A_{jk}x_kA_{jl}\delta_{il}= A_{ji}A_{jl}x_l+A_{jk}x_kA_{ji}=2A_{ij}^TA_{jk}x_k=(2A^TAx)_i,$$ so $\nabla\langle Ax,Ax\rangle=2A^TAx$.

Answer 2

Let's do it: first let's remember that $$\langle Ax, y \rangle = \sum_i (Ax)_iy_i = \sum_{i,j} A_{ij} x_j y_i = \sum_{i,j} x_j A^T_{ji} y_i = \langle x, A^T y \rangle$$

Thus $\langle Ax, Ax \rangle = \langle x, A^TA x \rangle$, following the suggestion of pppqqq let's write $A^TA =: B$. Then $\langle x, Bx \rangle = \sum_{i,j} b_{ij}x_ix_j$. Let's compute the gradient of this at a point $x$. Well, let's determine the $\alpha^{th}$ component, by definition this is: $$\partial_{\alpha} \sum_{i,j} b_{ij}x_ix_j = \sum_{j \neq \alpha} b_{\alpha j} x_j + \sum_{i \neq \alpha} b_{i \alpha} x_i + \sum_{i = j = \alpha} 2b_{\alpha \alpha} x_\alpha$$We may split up the last term into $b_{\alpha \alpha} x_\alpha + b_{\alpha \alpha} x_\alpha$, so we have$$= \sum_j b_{\alpha j} x_j + \sum_i b_{i \alpha} x_i = (Bx)_\alpha + \langle B(e_\alpha), x \rangle$$Now, we notice that in our case $B$ is its own transpose, so we have $$ = \sum_j b_{\alpha j}x_j + \sum_i b_{\alpha i} x_i = 2 \sum_j b_{\alpha j} x_j = 2(Bx)_\alpha$$as desired.

Answer 3

Expand it in each component, $$\begin{align}\left(\nabla\langle Ax,Ax\rangle\right)_i&=\partial_i\left(\sum_j\left(\sum_kA_{jk}x_k\right)\left(\sum_lA_{jl}x_l\right)\right)\\ &=\sum_j\left(A_{ji}\left(\sum_lA_{jl}x_l\right)+\left(\sum_kA_{jk}x_k\right)A_{ji}\right)\\ &=2\sum_jA_{ji}\left(\sum_kA_{jk}x_k\right)\end{align}$$ Or, $$\nabla\langle Ax,Ax\rangle=2A^TAx$$