How should I study The Matrix Cookbook?

Question

I use The Matrix Cookbook by Kaare Brandt Petersen and Michael Syskind Pedersen to solve many problems (mostly matrix derivatives). In most cases, I just map the problem to one of the formula and solve it but I cannot derive the formula by myself easily (I may prove the given formula is correct).

Since I do not have access to the book when I am taking test, I am wondering how others perform these kinds of calculation without a reference book. Is this book just considered as a reference or I should study the book and try to drive the formula by myself?

Does anyone have an insight on how to get better in matrix calculus (specially derivative with respect to vector or matrix) or how should I study such books? Thanks in advance.

Answer 1

The Matrix Cookbook is only a list of formulas without proof. It absolutely does not learn to understand the theory - and the practice - of derivation. Better, you can have a look at the solutions given in this website. Roll up your sleeves, my friend.

Let $g(U)=1/2tr(U^TU),f(W)=1/2tr((XW-T)^T(XW-T))$.

If $K$ has same dimensions as $U$, then $Dg_U(K)=1/2(tr(K^TU)+tr(U^TK))=tr(K^TU)$.

We deduce that, if $H$ has same dimensions as $W$, then $Df_W(H)=tr((XH)^T(XW-T))=tr(H^TX^T(XW-T))$.

If, for every $H$, $Df_W(H)=0$, then $X^T(XW-T)=0$, that implies

$W=(X^TX)^{-1}X^TT$ (if $X^TX$ is invertible).

Answer 2

I have the same problem!
I found two things help. 1) Knowing the basics and why the work the way they do. 2) Being able to prove things for myself.

I found 3 more references, that may be helpful. One of them helped me understand the basics specifically differentiation with respect to a vector and Hessian a lot better. https://www.sfu.ca/~haiyunc/notes/matrix_calculus.pdf It's short sweet and very nicely done reminder of the basics. The 2nd one I have not started reading... Matrix Calculus By Sourya Dey. Finally, the second reference also cites Matrix Calculus in Wikipedia and Wikipedia is almost always good https://en.wikipedia.org/wiki/Matrix_calculus