Matrix Differentiation proof II

Question

Someone asks a different part of this question here. I wonder how can we derive from (46) to (47)? Especially how to think the way to get the transpose?

Answer 1

I think the confusion might be coming from the row/column conventions for matrix calculus, which are often not well-explained in courses.

The important thing to remember is that matrix-vector multiplication can be defined per-component by $$ [Ax]_{i} = \sum_j a_{ij} x_j $$ And, for the transpose, we get $$ [A^Tx]_{i} = \sum_j a_{ji} x_j $$ So, (46) can be rewritten as: $$ \frac{\partial \alpha}{\partial x_k} = \sum_j a_{kj}x_j + \sum_i a_{ik}x_i = [Ax]_k + [A^Tx]_k $$ Clearly, the quantity $\frac{\partial \alpha}{\partial x}$ should be a vector with components $\frac{\partial \alpha}{\partial x_k}$. Now, we are looking at the Jacobian of $\alpha(x)=\sum_{i,j} a_{ij}x_ix_j$, so it should be a row vector. However, we have written the above as components of column vectors. So it is better to write: $$ \frac{\partial \alpha}{\partial x_k} = \sum_j a_{kj}x_j + \sum_i a_{ik}x_i = [x^TA^T]_k + [x^TA]_k $$ Note this makes no difference to the components since a transposed vector has the same components for a given index. However, now we correctly get a row vector when we generalize this as a matrix derivative: $$ \frac{\partial \alpha}{\partial x} = x^TA^T + x^TA = x^T(A^T+A) $$ by simply removing the component index.

Hopefully this way of looking at it helps!