I was going through my linear algebra notes and got a bit confused with the following: let $x$ be a vector in $\mathbb{R}^{n}$ and $A$ an $n\times n$ matrix, then $$ \frac{\partial x'A}{\partial x}=A $$ Since I was confused I tried the following "toy" example. Let $A$ be a $2\times 2$ matrix given by $$ A=\begin{bmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\end{bmatrix} $$ Thus $$ x'A=\begin{bmatrix}a_{11}x_{1}+a_{21}x_{2}&a_{12}x_{1}+a_{22}x_{2}\end{bmatrix} $$ Now how do I take "the derivative" of each element? Is such thing the Jacobian of the function $f(x_{1},x_{2})=\left(a_{11}x_{1}+a_{21}x_{2},a_{12}x_{1}+a_{22}x_{2}\right)$?
Thank you very much
Allright so I have checked the book "Matrix Algebra From a Statistician's Perspective" by Harville and I think I have reached an answer:
Let $\mathbf{f}=(f_{1},f_{2},\ldots,f_{n})'$ be a (column) vector function and $\mathbf{x}=\left(x_{1},x_{2},\ldots,x_{n}\right)'$. We define $\frac{d\mathbf{f}}{d\mathbf{x}'}$ as the Jacobian of such function, i.e. the $ij_{th}$ element is $\frac{df_{i}}{dx_{j}}$. In addition, we define $\frac{d\mathbf{f}'}{d\mathbf{x}}=\left(\frac{d\mathbf{f}}{d\mathbf{x}'}\right)'$ as the gradient matrix.
Thus, it is trivial to show that $\frac{d A\mathbf{x}}{d\mathbf{x}'}=A$ and therefore $\frac{d\mathbf{x}'A}{d\mathbf{x}}=\left(\frac{dA'\mathbf{x}}{d\mathbf{x}'}\right)'=\left(A'\right)'=A$.
In order to show that $\frac{dA\mathbf{x}}{d\mathbf{x}}=A'$ we know that $\frac{dA\mathbf{x}}{d\mathbf{x}}=\left(\frac{d\mathbf{x}'A'}{d\mathbf{x}'}\right)'$. The problem is that $\mathbf{x}'A$ is not a column of functions! What we need to do is to "redefine" it as the derivative of its transpose -$A\mathbf{x}$- in order to get a column (whose derivative we know how to compute). Since $\frac{dA\mathbf{x}}{d\mathbf{x}'}=A$ we conclude that $\frac{dA\mathbf{x}}{d\mathbf{x}}=A'$.