Derivative of matrix vector dot product with respect to matrix

Question

Given the function

$$f(N) = x_1^T M x_2 $$

where

• $x_1 = Nv_1 $
• $x_2 = Nv_2 $
• $x_1, x_2, v_1, v_2$ are vectors with dimension $n \times 1$
• $M$ and $N$ are matrices with dimension $n \times n$

what's the derivative of $f(N)$ with respect to $N$?

Answer 1

Generalizing the problem slightly, to use matrix (instead of vector) variables, $X_k = N\cdot V_k$.

We can write the function as $$ \eqalign { f &= X_1 : M\cdot X_2 \cr } $$ Now take the differential and expand $$ \eqalign { df &= dX_1 : M\cdot X_2 + X_1 : M\cdot dX_2 \cr &= d(N\cdot V_1) : M\cdot X_2 + X_1 : M\cdot d(N\cdot V_2) \cr &= dN\cdot V_1 : M\cdot X_2 + X_1 : M\cdot dN\cdot V_2 \cr &= dN : M\cdot X_2\cdot V_1^T + M^T\cdot X_1\cdot V_2^T : dN \cr &= [M\cdot X_2\cdot V_1^T + M^T\cdot X_1\cdot V_2^T] : dN \cr &= [M\cdot N\cdot V_2\cdot V_1^T + M^T\cdot N\cdot V_1\cdot V_2^T] : dN \cr } $$ The derivative is therefore $$ \eqalign { \frac {\partial f} {\partial N} &= M\cdot N\cdot V_2\cdot V_1^T + M^T\cdot N\cdot V_1\cdot V_2^T \cr } $$ If you dislike the Frobenius product, you can change the above derivation to use the trace instead $$ \eqalign { A : B &= \text{tr}(A^T\cdot B) \cr } $$

Answer 2

Rephrasing, let

$$ f ({\bf X}) := {\bf b}_1^\top {\bf X}^\top {\bf A} \, {\bf X} \, {\bf b}_2 = \operatorname{tr} \left( {\bf X}^\top {\bf A} \, {\bf X} \, {\bf b}_2 {\bf b}_1^\top \right) $$

Taking the gradient,

$$ \nabla_{{\bf X}} f ({\bf X}) = \cdots = \color{blue}{{\bf A} \, {\bf X} \, {\bf b}_2 {\bf b}_1^\top + {\bf A}^\top {\bf X} \, {\bf b}_1 {\bf b}_2^\top} $$

---

Related

• How is $\nabla_X \operatorname{Tr}\left\{ X^T A X B\right\} = AXB + A^TXB^T$?