Given matrix $A$, I would like to compute the following derivative with respect to matrix $X$.
$$\frac{\partial}{\partial X} \left( \frac{AXA}{\operatorname{Tr} \left( AXA \right)} \right) $$
I know that the derivative of the numerator is a fourth order tensor and the derivative of the denominator is a matrix, but I'm not sure how to combine them for the derivative of the quotient. Would you know how to do it?
$ \def\p{\partial} \def\b{\beta} \def\E{E_{k\ell}} \def\F{F_{ij}} \def\X{X_{k\ell}} \def\LR#1{\left(#1\right)} \def\BR#1{\Big(#1\Big)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\qiq{\quad\implies\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\fracLR#1#2{\LR{\frac{#1}{#2}}} $Define the variables $$\eqalign{ B &= AXA &\qiq dB = A\:dX\,A \\ \b &= \trace B &\qiq d\b = \trace{A\:dX\,A} \\ }$$ and the matrix $\E$ whose elements are all zero except for a ${\tt1}$ in the $(k,\ell)$ position.
Then the function of interest is $$\eqalign{ F=\b^{-1}B \qquad\qquad\qquad\qquad\qquad\qquad }$$ and its component-wise gradients can be calculated as $$\eqalign{ dF &= \b^{-1}dB - B\,\b^{-2}d\b \\ &= \b^{-2}\BR{\b\:dB - B\:d\b} \\ &= \b^{-2}\BR{\b\,A\:dX\,A - B\,\trace{A\:dX\,A}} \\ \\ \grad{F}{\X} &= \b^{-2}\BR{\b\,A\E A - B\,\trace{A\E A}} \\ \\ \grad{\F}{\X} &= \b^{-2}\BR{\b\,\LR{A\E A}_{ij} - B_{ij}\trace{A\E A}} \\ }$$ Note that there are four indexes, so the gradient is a fourth order tensor as you anticipated.