What is the derivative of a matrix w.r.t itself?

Question

what is the derivative of \begin{equation}\partial \frac{x^TVx}{\partial V} \end{equation} where V is a matrix and x is a vector. In general what is the right way to calculate matrix derivatives w.r.t other matrices or vectors?

I have seen this and this but have no clue how they work.

Answer 1

You want to differentiate a scalar quantity $x^TVx$ with respect to matrix $V$, so that the derivative will be a matrix with the same dimension as $V$.

Now, $x^TVx$ is equal to $Trace(Vxx^T)$, so using standard results of the derivative of the trace of a matrix product, see page $3$ here, the result is $$\frac{\partial x^TVx}{\partial V}=\frac{\partial Trace(Vxx^T)}{\partial V}=xx^T$$

Answer 2

alternately, recall that a derivative is the best approximating linear map. Also the map $$ V \mapsto x^{t}Vx $$ is linear in $V$ so we can expect to get what we started with.

Consider a small perturbation $E$ $$ x^{t} (V+E) x^t =x^t V x + x^t E x $$ So the best approximation for local behaviour around $V$ is the map $$ E \mapsto x^{t} E x. $$