Derivative Calculation of vector and matrix with transpose

Question

How to calculate the derivative of $f(x)= v^T(Ax)$? I have seen an equation in which it is written as $f^{'}(x)=A^Tv$. I am not understanding that why the transpose is interchanged. Here v is a vector$ R^{p*1}$ and A is a matrix $R^{p*n}$

Answer 1

$\newcommand{\R}{\Bbb{R}}$ Use the following fact:

If $f(x) = cx$ where $c\in \R^{1\times n}$ is a constant row vector and $x \in\R^n$, then $f'(x) = c^T \in \R^n$.

In your case, $c = v^TA$, so the derivative is $c^T = (v^TA)^T = A^Tv$.

---

To see why that fact is true, note that $f'(x)$ refers to the column vector whose $i$-th component is $\frac{\partial f}{\partial x_i}$ for all $i=1,\ldots,n$ (and $x_i$ is the $i$-th component of $x$). Write $c = \begin{bmatrix}c_1 & c_2 & \cdots & c_n\end{bmatrix}$ and $x = \begin{bmatrix}x _1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix}$. We have $f(x) = cx = c_1x_1 + c_2 x_2 + \cdots +c_n x_n$, so $\frac{\partial f}{\partial x_i} = c_i$. Hence for each $i=1,\ldots,n$, the $i$-th component of $f'(x)$ is $c_i$. This means that $f'(x)$ (written as a column vector) is just $\begin{bmatrix}c _1 \\ c_2 \\ \vdots \\ c_n\end{bmatrix} = c^T$.

Answer 2

The transposition is a matter of convention. Usually people write the gradient as a column vector. $v^TA$ will have only one row, so it is a row vector. $A^Tv$ will have the same elements, but as a column vector. The convention is important only because we must be consistent. This is discussed (at a somewhat advanced level) in the Wikipedia page on matrix calculus:

https://en.wikipedia.org/wiki/Matrix_calculus#Layout_conventions