I am trying to differentiate the following function, with respect to a matrix $X$:
$$ \operatorname{tr}(AX(X^TX)^{-1}B) $$
where tr corresponds to the trace. Is there an easy way to see what the derivative will be? I've come across rules for $\operatorname{tr}(AXB)$ but not the form above with inverses etc .
$A$ and $B$ are known (constant) matrices.
Define $W \equiv (X'X)^{-1}$, then $f = BAX : W$ and its derivative is $$ \frac {\partial f} {\partial X} = A'B'W - XW(BAX+X'A'B')W $$ The algebra to arive at this result is tedious but straight-forward. The only tricky part is knowing that $$ \eqalign { dW &= - W\,\,dY\,\,W \cr } $$ where $Y \!\equiv W^{-1}\!= X'X$
It's worth noting that both $Y$ and $W$ are symmetric.
Then you just expand the differential of $f$
$$ \eqalign{ df &= BA\,dX : W + BAX : dW \cr &= dX : (BA)'W - BAX : W\big[dX'X + X'dX\big]W \cr &= dX : A'B'W - WBAXW : \big[dX'X + X'dX\big] \cr &= A'B'W : dX - WBAXW : \big[dX'X + X'dX\big] \cr &= A'B'W : dX - WBAXW : dX'X - WBAXW : X'dX \cr &= A'B'W : dX - WBAXWX' : dX' - XWBAXW : dX \cr &= A'B'W : dX - XWX'A'B'W : dX - XWBAXW : dX \cr &=[A'B'W - XWX'A'B'W - XWBAXW]: dX \cr } $$ So the expression in brackets must be the derivative.
If you dislike the Frobenius product, you can carry out the above steps using the trace $$ \text{tr}(A'B) = A:B $$