What is the matrix derivative of a symmetric bilinear form $\mathbf{a}^T X \mathbf{b}$ wrt $X$?

Question

What is the derivative of the symmetric bilinear form $$ f_X(\mathbf{a},\mathbf{b}) = \mathbf{a}^T X \mathbf{b} $$ with respect to the (symmetric) matrix $X$?

Following Wikipedia, and using the denominator layout, I would say $$ \frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \mathbf{a}\mathbf{b}^T $$

But since $f_X$ is symmetric, $f_X(\mathbf{a},\mathbf{b}) = f_X(\mathbf{b},\mathbf{a})$. If I derive this equality I obtain $$ \frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \frac{\partial f_X}{\partial X}(\mathbf{b},\mathbf{a}) \qquad\Rightarrow\qquad \mathbf{a}\mathbf{b}^T = \mathbf{b}\mathbf{a}^T $$ which is wrong because $\mathbf{a}\mathbf{b}^T \neq \mathbf{b}\mathbf{a}^T$

Where am I wrong?

Answer 1

Wikipedia has calculated the gradient with respect to an unconstrained $X$ matrix, i.e. $$G = ab^T$$ To enforce the symmetry constraint, the result should be symmetrized $$\eqalign{ G_{sym} &= \tfrac 12(G+G^T) \;=\; \tfrac 12(ab^T + ba^T) }$$

Answer 2

The problem is that the so-called 'denominator layout'

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b}) = \mathbf{a}\mathbf{b}^T$$

is just a very special shorthand notation for the general directional derivative

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \mathbf{a}^TH\mathbf{b}$$

where $H$ is the direction into which you are deriving, i.e. a general operator between the source and destination spaces. In particular, $H$ is not restricted to being a symmetric operator!

You can see the connection between directional derivative and 'denominator layout' symbolism by writing the bilinear form as a trace. The trace can be permuted cyclically (first step) and is invariant under transposition of its argument (second step):

$$\mathbf{a}^TH\mathbf{b}=\rm{tr}(H\mathbf{b}\mathbf{a}^T)=\rm{tr}(\mathbf{a}\mathbf{b}^TH^T)$$

Hence, the 'denominator layout' matrix derivative is the kernel $\mathbf{a}\mathbf{b}^T$ of the trace contained in the directional derivative with respect to $H^T$. No surprise, you were getting a little confused...

If you are sticking closely to the directional derivative, everything is clearer. Since your very special direction $H$ is not an arbitrary operator, but a symmetric one, according to your presumptions, we have

$$H=\frac{1}{2}(H+H^T)$$

and hence,

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \frac{1}{2}(\mathbf{a}^T(H+H^T)\mathbf{b})=\frac{1}{2}\mathbf{a}^TH\mathbf{b}+\frac{1}{2}\mathbf{b}^TH\mathbf{a}$$

where in the last step the invariance under transposition has been used again. So, finally, we obtain the directional derivative of your bilinear form for the special case of a symmetric $X$ and $H$ respectively:

$$\frac{\partial f_X}{\partial X}(\mathbf{a},\mathbf{b})[H] = \rm{tr}\left(\frac{1}{2}\left(\mathbf{b}\mathbf{a}^T+\mathbf{a}\mathbf{b}^T\right)H\right)$$

More specifically, the question 'Where am I wrong?' can now be answered:

You used the symmetry constraint $X^T=X$ when you computed the derivative, but you did not apply the symmetry constraint to the (hidden, in your preferred notation) directional argument $H$ as well. Instead, by not paying attention to $H$, you have treated it as if it was still a completely general operator. You were vulnerable to this issue because the 'denominator layout' notation omits the direction $H$ into which you actually derive.

By applying the constraint to the directional argument as well, you would have obtained the derivative in the symmetrized form in the first place, as you can see above, and not in the unsymmetric form that leads to the false conclusion

$$\mathbf{b}\mathbf{a}^T=\mathbf{a}\mathbf{b}^T.$$

Answer 3

The partial (i.e. unconstrained) derivative $$\frac{\partial f(X)}{\partial X}=Y$$ is given by

$$Y_{i,j}= \frac{\partial f(X)}{\partial X_{i,j}}$$

For $f(X)=a^T X b =\sum_{i,j} a_i X_{i,j} b_j$ this gives $Y_{i,j}=a_i b_j$ or $Y= a b^T$.

But here we are interested in the total derivative. Because the matrix is symmetric we have the restriction $dX_{i,j} = dX_{j,i}$ and

$$d f(X)= \begin{cases} \frac{\partial f(X)}{\partial X_{i,j}} d X_{i,j} + \frac{\partial f(X)}{\partial X_{j,i}} d X_{j,i} & i\ne j\\ \frac{\partial f(X)}{\partial X_{i,j}} d X_{i,j} & i=j \end{cases} $$

Hence in our case

$$ \frac{d f(X)}{dX_{i,j}}= \begin{cases} \frac{\partial f(X)}{\partial X_{i,j}} + \frac{\partial f(X)}{\partial X_{j_,i}} = a_i b_j + b_i a_j & i\ne j\\ \frac{\partial f(X)}{\partial X_{i,j}} = a_i b_i & i=j \end{cases} $$

This can be written as

$$\frac{d f(X)}{dX}= a^T b + b^T a - diag(a \circ b)$$

See also here