Matrix differential and differentation

Question

I need to find the differential and derivative of $ f: X\rightarrow (I-X(X'X)^{-1}X')$

Now by the product rule I found that the differential of $d(I-X(X'X)^{-1}X')=-(dX(X'X)^{-1}X' + (X)d(X'X)^{-1}X'+X(X'X)^{-1}d(X')$

Now I have issues with finding $d(X'X)^{-1}$ because I am not sure how to use the chain rule here, and with the first term: $(dX(X'X)^{-1}X'$ because it is not in the standard form $A(dX)B$ where $A$ and $B$ are any matrices.

Could anyone show me how to proceed as to find the derivative of this function?

Answer 1

For convenience, define $M=(X^TX)^{-1}X^T\,\,$ then the function is $$\eqalign{ F &= I-XM = \,F^T\cr }$$ And the differentials are $$\eqalign{ dM &= (X^TX)^{-1}dX^T - (X^TX)^{-1}(dX^TX)(X^TX)^{-1}X^T - (X^TX)^{-1}(X^TdX)(X^TX)^{-1}X^T \cr &= (X^TX)^{-1}dX^T - (X^TX)^{-1}\,dX^T\,XM - M\,dX\,M \cr &= (X^TX)^{-1}dX^T\,F - M\,dX\,M \cr \cr dF &= -X\,dM - dX\,M \cr &= -X(X^TX)^{-1}\,dX^T\,F+XM\,dX\,M \,-\, dX\,M\cr &= -M^T\,dX^T\,F^T - F\,dX\,M \cr }$$ Apply ${\rm vec}()$ to both sides, but remember that $\,{\rm vec}(dX^T) \neq dx^T$ even though $\,{\rm vec}(dX) = dx$.

Instead, you must use the Kronecker permuation matrix $P$, which depends on the matrix dimensions, to obtain $\,{\rm vec}(dX^T) = P\,dx$. $$\eqalign{ df &= -(F\otimes M^T)\,P\,dx - (M^T\otimes F)\,dx \cr \frac {\partial\,{\rm vec}(F)}{\partial\,{\rm vec}(X)^T} =\frac {\partial f}{\partial x^T} &= -(F\otimes M^T)\,P - (M^T\otimes F) \cr }$$ This kind of vec/vec result is typical for matrix-by-matrix derivatives.

Answer 2

It suffices to derive a product of $3$ terms. Recall that a derivative is a linear function. Let $X\in M_{n,m}$. Then

$Df_X:H\in M_{n,m}\rightarrow -H(X^TX)^{-1}X^T+X(X^TX)^{-1}(H^TX+X^TH)(X^TX)^{-1}X^T-X(X^TX)^{-1}H^T$

We cannot write this expression in tensor form except if we use the function $H\rightarrow H^T$.