I was interested in seeing how the normal equations for least squares for linear regression are derived, and found this page: https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression/
I am fine with most of the derivation except for two key lines.
They derive the loss function as:
And then simplify this to:
I don't understand how $(X\theta)^T y - y^T (X\theta)$ simplifies to $-2(X\theta)^T y$.
Notice that $-(X\theta)^T y = -(X\theta) \cdot y$ where $\cdot$ is the dot product. Similarly, $-y^T(X\theta) = -y \cdot (X\theta) = -(X\theta) \cdot y$ (since the dot product is symmetric).
Hence, the sum of these two values is $-2(X\theta)\cdot y = -2(X\theta)^Ty$