Partial derivative of $x^TA^TAx$ with respect to $A$

Question

I'm trying to evaluate $\nabla_{A} x^TA^TAx$ for vector $x \in \mathbb{R}^n$, and matrix $A \in \mathbb{R}^{m \times n}$. I evaluated it elementwise and was wondering if there is a clean, closed form formula for this?

Answer 1

$ \def\o{{\tt1}}\def\p{\partial} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} $The Matrix Cookbook is a wonderful resource, but you may want to verify the result for yourself.

The matrix inner product is a convenient notation for the trace $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \big\|A\big\|^2_F \\ }$$ When applied to vector arguments, it corresponds to the usual dot product.

For typing convenience, introduce the vector $\;w=Ax$.

Write the objective function using the above notation.
Then calculate its differential and gradient. $$\eqalign{ \phi &= \big\|Ax\big\|^2_F = w:w \\ d\phi &= 2w:dw = 2w:dA\,x = 2wx^T:dA \\ \grad{\phi}{A} &= 2wx^T = 2Axx^T \\ }$$