What's the derivative of $X^TX$ w.r.t. $X$

Question

What's the derivative of $X^TX$ w.r.t. $X$ Note that X is not a square matrix

Answer 1

For any Matrix $H$ of the same dimension as $X$ the following holds:

$$(X + H)^t (X + H) = X^tX + H^tX + X^tH + H^tH$$ Since $H^tH = o(H)$ for $H \to 0$, this means the derivative of the map at $X$ is given by the linear map $H \mapsto H^tX + X^tH$.