For Hermitian matrices $A$ and $X$, I'm wondering how to calculate $\frac{\partial \operatorname{tr}(AXAX)}{\partial X}$.
My attempt:
Let $f(X) \mapsto \operatorname{tr}(AXAX)$. Expanding $f(X+H) = \operatorname{tr}(AXAX) + 2\operatorname{tr}(AXAH) + \operatorname{tr}(AHAH)$. Since the second term is $\operatorname{tr}((2AXA)^\dagger H)$ corresponds to the inner product of the gradient and $H$, the gradient seems to be $2AXA$. Is this a right approach?
Let $M=AX,\,$ then applying the following formula from the Matrix Cookbook $$\eqalign{ {d\operatorname{Tr}(M^k)} = (kM^{k-1})^T:dM \\ }$$ for $k=2\,$ and $\,dM=A\,dX\;$ yields $$\eqalign{ {d\operatorname{Tr}(M^2)} &= 2M^T:A\,dX \;=\; 2A^TM^T:dX \\ \frac{\partial \operatorname{Tr}(M^2)}{\partial X} &= 2A^TM^T \;=\; 2(AXA)^T \\ }$$ ...or the transpose of this, depending on your preferred layout convention.