The function is :$f(X)$=Trace(1$^{T}_{n\times m}$ $\sqrt{(AX)^2+(XB^T)^2}$)
1$_{n\times m}$ is a matrix with all the elements equal to 1.
$A$ is an $n \times n$ matrix. $X$ is an $n \times m$ matrix. $B$ is an $m \times m$ matrix
How to compute d$f(X)$/d$X$?
In $\sqrt{(AX)^2+(XB^T)^2}$, the square root and square are defined element-wise.
Denote the elementwise square-root as $S$ so that
$$\eqalign{ S\odot S &= (AX\odot AX) + (XB^T\odot XB^T) \\ }$$ and assume that its elementwise inverse $P$ exists, such that $\,P\odot S = {\tt1}$
Then the desired gradient is $$\eqalign{ \frac{\partial f}{\partial X} &= A^T(AX\odot P) + (P\odot XB^T)B \\ \\ }$$ Here is how this result was derived.
Let a colon denote the trace/Frobenius product, $$A:B = {\rm Tr}(A^TB)$$ The properties of the trace allow terms in a Frobenius product to be rearranged in many ways, e.g. $$\eqalign{ A:BC &= AC^T:B \\&= B^TA:C \\&= I:A^TBC \\ }$$ Note that the Frobenius and Hadamard products commute with themselves and each other. $$\eqalign{ A:B &= B:A \\ A\odot B &= B\odot A \\ A:B\odot C &= A\odot B:C \\ }$$ Start by taking the differential of the square-root relation $$\eqalign{ 2S\odot dS &= 2(AX\odot A\,dX) + 2(dX\,B^T\odot XB^T) \\ }$$ Next, use these new variables to write the function.
Then calculate its differential and gradient. $$\eqalign{ f &= {\tt1}:S \\ df&= {\tt1}:dS \\ &= (P\odot S):dS \\ &= P:(S\odot dS) \\ &= P:(AX\odot A\,dX) + P:(dX\,B^T\odot XB^T) \\ &= (AX\odot P):A\,dX + (P\odot XB^T):dX\,B^T \\ &= A^T(AX\odot P):dX + (P\odot XB^T)B:dX \\ \frac{\partial f}{\partial X} &= A^T(AX\odot P) + (P\odot XB^T)B \\ }$$