What's gone wrong in my example to show that $\frac{\partial}{\partial \underline c}(A\underline c) = A$?

Question

First, an example:

Suppose $A$ is a symmetric matrix $A = \begin{bmatrix} a & b \\ b & a \\ \end{bmatrix}$, and $\underline c = \begin{bmatrix} c_1 \\ c_2 \end{bmatrix}$

Then $A \underline c = \begin{bmatrix} ac_1 + b c_2 \\ bc_1 + ac_2 \\ \end{bmatrix}$

and hence, $\frac{\partial}{\partial \underline c}(A\underline c) = \begin{bmatrix} \frac{\partial}{\partial c_1}(ac_1 + b c_2) \\ \frac{\partial}{\partial c_2}(bc_1 + ac_2) \\ \end{bmatrix} = \begin{bmatrix} a \\a \end{bmatrix} \neq A$

However, I'm told that the derivative with respect to the vector of the product of a symmetric $(n\times n)$ matrix $A$ and an $(n\times 1)$ vector $\underline c$ is equal to the matrix $A$, such that,

$\frac{\partial}{\partial \underline c}(A\underline c) = A$

Where is the error in my understanding?

Answer 1

You have a vector result $$ y = Ac$$

Thus, you are computing a vector-by-vector gradient which results in a second-order tensor, ie. a matrix, with the following form:

$$ \frac{\partial y}{\partial c}= \begin{bmatrix} {\frac {\partial y_{1}}{\partial c_{1}}}&{\frac {\partial y_{1}}{\partial c_{2}}}\\ {\frac {\partial y_{2}}{\partial c_{1}}}&{\frac {\partial y_{2}}{\partial c_{2}}}\\ \end{bmatrix} $$

You are just missing the off-diagonal terms.

Answer 2

Your definition of the derivative $ \frac{\partial}{\partial \overline{c}}. $

The function $$\overline{c} \mapsto A \overline{c}$$ is total differentiable if there exists a linear map $B$, with $$ \lim_{\|h\| \rightarrow 0} \frac{\|A\left(\overline{c}+\overline{h}\right) - A\overline{c} - B\overline{h}\|}{\|\overline{h}\|} = 0 $$.

Since the map is linear, it is obvious that $A = B$, so the total derivative is $A$.

Answer 3

Since your function is vector valued, its derivative is a matrix : $\left(\frac{\partial \mathbf{u}}{\partial\mathbf{x}}\right)_{i,j} = \frac{\partial u_i}{\partial x_j}$.

Hence, $\frac{\partial A \mathbf{c}}{\partial \mathbf{c}} = \begin{bmatrix} \frac{\partial a c_1 + b c_2}{\partial c_1} & \frac{\partial a c_1 + b c_2}{\partial c_2} \\ \frac{\partial b c_1 + a c_2}{\partial c_1} & \frac{\partial b c_1 + a c_2}{\partial c_2}\end{bmatrix} = \begin{bmatrix} a & b\\ b & a\end{bmatrix}$.

Remark : in general, $\frac{\partial A \mathbf{c}}{\partial \mathbf{c}} = A^T $, but here your matrix is symmetric (i.e. $A=A^T$).