Confusion with matrix derivatives and component-wise gradients

Question

According to this source (The Matrix Cookbook), we have that for $f(X) = \mathbf{a}^T X \mathbf{b}$ where $X \in \mathbb{R}^{n \times m}, \mathbf{a} \in \mathbb{R}^n, \mathbf{b} \in \mathbb{R}^m$, the derivative is $$ \frac{\partial f}{\partial X} = \mathbf{a} \mathbf{b}^T$$ which is a scalar since it's basically a dot product. I see how we can arrive at this result by applying matrix derivative rules/applying basic derivative rules to vectors/matrices. However, taking the derivative (gradient) of a function $f: \mathbb{R}^{n \times m} \to \mathbb{R}$ based on my my calculus knowledge should yield a vector/matrix of partial derivatives.

As an example, I tried this with $\mathbf{a} = \begin{bmatrix} a_1 & a_2 \end{bmatrix}, \mathbf{b} = \begin{bmatrix} b_1 & b_2 \end{bmatrix}$, and $X = \begin{bmatrix} x_1 & x_2 \\ x_3 & x_4 \end{bmatrix}$. I expanded the expression and took the gradient to get that $$\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} \\ \frac{\partial f}{\partial x_3} & \frac{\partial f}{\partial x_4} \end{bmatrix} = \begin{bmatrix} a_1b_1 & a_1b_2 \\ a_2b_1 & a_2b_2 \end{bmatrix}.$$

So, my question is, is this the derivative, or is it $\mathbf{a}\mathbf{b}^T$? Should the derivative be a vector or a scalar?

P.S. I also noticed that $\mathbf{a}\mathbf{b}^T = \operatorname{trace}(\nabla f)$ for the matrix I computed for $\nabla f$. Is there any significance to that?

Answer 1

The problem is vector usually refers to a column vector (otherwise the expression $a^TXb$ doesn't even make sense since the dimensions don't match). If you instead do $$ab^T = \begin{pmatrix}a_1\\a_2\end{pmatrix}\begin{pmatrix}b_1 & b_2\end{pmatrix} = \begin{pmatrix} a_1b_1 & a_1b_2\\ a_2b_1 & a_2b_2 \end{pmatrix}$$ then you get the same result as your computation.

Answer 2

There are several competing notations. The notation in Matrix Cookbook can be motivated as follows -- suppose we minimize scalar function of scalar variable $w$ using gradient descent update

$$w\leftarrow w-f'(w)$$

Now, suppose $f(W)$ is a scalar function of matrix variable. We want similar looking update formula to hold

$$W\leftarrow W-f'(W)$$

So this determines the shape of $f'$: $i,j$th entry of $f'$ is the derivative of $f$ w.r.t. to $i,j$the entry of $W$, which gives you $f'(W)=ab^T$ when $f(W)=a'Wb$

Answer 3

The Jacobian matrix is the derivative of a vector function $\pmb{f}$ of a vector $\pmb{x}$: $$ \frac{\partial \pmb{f}}{\partial \pmb{x}^T} = \begin{bmatrix}\frac{\partial f_1}{\partial x_{1}} & \frac{\partial f_1}{\partial x_{2}} \\ \frac{\partial f_2}{\partial x_{1}} & \frac{\partial f_2}{\partial x_2} \end{bmatrix} $$ i.e. the partial derivatives are w.r.t. the transpose of the vector. The relation between the differential of $\pmb{f}$ and the differential of $\pmb{x}$ is, \begin{align*} d\pmb{f}= \begin{bmatrix} df_1 \\ df_2 \end{bmatrix} & = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} dx_1 + \frac{\partial f_1}{\partial x_2} dx_2 \\ \frac{\partial f_2}{\partial x_1} dx_1 + \frac{\partial f_2}{\partial x_2} dx_2 \end{bmatrix} \\ &= \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} \\ \frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} \end{bmatrix} \begin{bmatrix} dx_1 \\ dx_2 \end{bmatrix} =\frac{\partial \pmb{f}}{\partial \pmb{x}^T} d\pmb{x} \end{align*} which shows that the definition of the derivative of $\pmb{f}$ w.r.t. $\pmb{x}$ as $\frac{\partial \pmb{f}}{\partial \pmb{x}^T}$ is correct.

In the same way given $f=\pmb{a}^T\pmb{X}\pmb{b}$ we have, $$ f = \begin{bmatrix}a_1 & a_2 \end{bmatrix} \begin{bmatrix}X_{11} & X_{12} \\ X_{21} & X_{22} \end{bmatrix} \begin{bmatrix}b_1 \\ b_2 \end{bmatrix} $$ and, $$ \frac{\partial f}{\partial \pmb{X}^T} = \begin{bmatrix}\frac{\partial f}{X_{11}} & \frac{\partial f}{X_{21}} \\ \frac{\partial f}{X_{12}} & \frac{\partial f}{X_{22}} \end{bmatrix} $$ i.e. the partial derivatives are w.r.t. the transpose of the matrix. Since, $$ f=a_1 (b_1X_{11}+b_2X_{12}) +a_2(b_1X_{21}+b_2X_{22}) $$ we have, \begin{align*} \frac{\partial f}{\partial X_{11}} &= a_1 b_1 & \frac{\partial f}{\partial X_{21}} &= a_2 b_1 \\ \frac{\partial f}{\partial X_{12}} &= a_1 b_2 & \frac{\partial f}{\partial X_{22}} &= a_2 b_2 \end{align*} and so, $$ \frac{\partial f}{\partial \pmb{X}^T} = \begin{bmatrix} a_1b_1 & a_2b_1 \\ a_1b_2 & a_2b_2 \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} \begin{bmatrix} a_1 & a_2 \end{bmatrix}= \pmb{b}\pmb{a}^T $$

The differential of $f$, \begin{align*} df & = \frac{\partial f}{\partial X_{11}} d X_{11} + \frac{\partial f}{\partial X_{12}} d X_{12} + \frac{\partial f}{\partial X_{21}}d X_{21} + \frac{\partial f}{\partial X_{22}}d X_{22} \\ & =a_1b_1 dX_{11} + a_1b_2 dX_{12} +a_2b_1 dX_{21} + a_2b_2 dX_{22} \\ & =\mathrm{tr} \left( \begin{bmatrix} a_1b_1 & a_2b_1 \\ a_1b_2 & a_2b_2 \end{bmatrix} \begin{bmatrix} dX_{11} & dX_{12} \\ dX_{21} & dX_{22} \end{bmatrix} \right) \\ & = \mathrm{tr} \left(\frac{\partial f}{\partial \pmb{X}^T} d\pmb{X} \right) \end{align*} and so defining the derivative of $f$ w.r.t. $\pmb{X}$ as above generates the correct differential.