let's begin with the simple case where $A$ is just a $2\times2$ matrix
let $\begin{align} f :& \mathbb{R^{2\times2}} \to \mathbb{R} \\ & A \mapsto \det(A) \end{align}$
I want to find the differential of this mapping if $A$ is invertible.
as a hint I was suggested to compute the following limit :
$\lim_{t \to 0} \frac1t [\det(I+tX) -1]$ where $X \in \mathbb{R^{2\times2}}$
the limit turns out to be just the trace of $X$
and since $\det(I) = 1 $ we have that $Df(I)(X) = Tr(X)$, right ?
so I guess now that if I want to find $Df(A)(X)$ for $A$ invertible I have to compute this limit :
$\lim_{t \to 0} \frac1t [\det(A+tX) -\det(A)] = \det(A)\lim_{t \to 0} \frac1t [\det(I+tA^{-1}X) -1] $
so $Df(A)(X) = \det(A)Tr(A^{-1}X)$, right ?
now in higher dimensions the last step wouldn't change and I guess that in the first limit the expression $[\det(I+tX) -1]$ would be something of the form $tTr(X) +t^2(\cdots) + t^3(\cdots)+\cdots$
so it's all cool but what if $A$ is not invertible ? the $\det$ being some sort of a polynomial would still be differentiable, right ? but how do you construct the differential in this case ?
Edit : my bad if $A$ is not invertible then $\det(A) = 0$ so I guess $Df(A)(X) = 0$ ? can someone confirm this ?
Edit 2 : at the end it all comes down to evaluating this : $$\lim_{t \to 0 } \frac1t \det(A+tX)$$ for $A$ non-invertible and $X \in \mathbb{R^{n\times n}}$
If you don't insist in having an explicit formula in terms of matrix operations, there is a nice way to evaluate the derivative in terms of the colums of matrices. If you write $A=(a_1,\dots,a_n)$ and $X=(x_1,\dots,x_n)$ you can compute $D\det(A)(X)$ as $$\tfrac{d}{dt}|_{t=0}\det(A+tX)=\tfrac{d}{dt}|_{t=0}\det(a_1+tx_1,\dots,a_n+tx_n). $$ Using multilinearity of the determinant, you can directly expand this and invertibility plays no role in the whole computation, to get $\sum_{i=1}^n\det(a_1,\dots,a_{i-1},x_i,a_{i+1},\dots,a_n)$. In particular, you see readily from this formula that $Df(A)(X)$ vanishes for all $X$ if and only if the rank of $A$ is at most $n-2$.