Let $\mathbb{M}^n$ be the set of real $n \times n$ matrices, and let $A$ be a fixed real $n \times n$ matrix. Define the function $F: \mathbb{M}^n \rightarrow \mathbb{R}$ by $$ F(X) = \det( A^T (I-X) A) $$ What is the Fréchet derivative (total derivative) of $F$ at a matrix $X$?
I am having difficulty working it out explicitly. I know that we can apply a chain rule, as $F = G \circ H$ where $H(X) = A^T(I-X)A$ and $G(Y) = \det(Y)$. Yet, I am confused how to apply the chain rule here. I am looking for an expression for the Fréchet derivative and a derivation would be wonderful as well.
Let $$Y = (I-X),\quad\alpha=\det A$$ Then $$\eqalign{ F &= \alpha^2\det Y \cr dF &= \alpha^2(\det Y)\,(Y^{-1})^T:dY \,\,= F\,Y^{-T}:(-dX) \cr \frac{\partial F}{\partial X} &= -FY^{-T} = \big(X^T-I\big)^{-1}\,\det\big(A^T(I-X)A\big) \cr }$$ where colon denotes the trace inner product, i.e. $\,A:B={\rm Tr}(A^TB)$
Update
If $A$ is rectangular, then $$\eqalign{ Y &= A^T(I-X)A \cr F &= \det Y \cr dF &= (\det Y)\,Y^{-T}:dY \,\,= -F\,Y^{-T}:A^TdX\,A \cr \frac{\partial F}{\partial X} &= -(AY^{-T}A^T)\,\,F = \Big(A\big(A^T(X^T-I)A\big)^{-1}A^T\Big)\,\,\det\big(A^T(I-X)A\big) \cr }$$