Ordinary Least Squares Derivative

Question

I have been trying to follow the derivation of the normal equations, but there is one part I do not understand.

So, if we minimize

$L(\mathbf{b})=\mathbf{y}^T\mathbf{y}-(2\mathbf{y}^T\mathbf{X})\mathbf{b}+\mathbf{b}^T(\mathbf{X}^T\mathbf{X})\mathbf{b}$

then $\frac{\delta L(\mathbf{b})}{\delta \mathbf{b}}= \mathbf{0}-2\mathbf{X}^T\mathbf{y}+2(\mathbf{X}^T\mathbf{X})\mathbf{b}$

I would have thought $(2\mathbf{y}^T\mathbf{X})\mathbf{b}$ simply becomes $(2\mathbf{y}^T\mathbf{X})$. But apparently it does not, and I cannot find the full derivation anywhere. I'd be very grateful for an explanation.

Answer 1

There are two ways of writing it - in either way you must make sure you are consistent with where the index of your derivative goes.$$\frac{dL}{db_p}=\frac{d}{db_p}\left( y_jy_j-2y_iX_{ij}b_j+b_i X_{ki}X_{kj}b_j \right)=0-2y_iX_{ip}+X_{kp}X_{kj}b_j+b_iX_{ki}X_{kp}\\=-2y_iX_{ip}+2X_{ki}X_{kp}b_i$$This can be written in one of two ways: $\left[-2y^TX+2b^TX^TX \right]_p$ or $\left[-2X^Ty+2X^TXb \right]_p$. The former is in the form of a row vector, the latter is a column vector. You probably want your answer to be a column vector, so you go for $$\frac{dL}{d\vec b}=-2X^T\vec y+2X^TX\vec b$$

Answer 2

(Since everything I'm gong to talk about would be bold, I'm not going to bother.)

Did you try the $2 \times 2$ version to get some insight? $$ y = \begin{pmatrix}y_1 \\ y_2 \end{pmatrix}, X = \begin{pmatrix} x_{11} & x_{12} \\ x_{21} & x_{22}\end{pmatrix} \text{.} $$ So, $$2 X^T y = \begin{pmatrix}2x_{11}y_1 + 2x_{21}y_2 \\ 2x_{12}y_1 + 2x_{22}y_2 \end{pmatrix}$$ and $$2 y^T X = \begin{pmatrix}2x_{11}y_1 + 2x_{21}y_2 & 2x_{12}y_1 + 2x_{22}y_2 \end{pmatrix} \text{.}$$ They're the same thing, up to transposition.