I'm trying to find
$\max_X \text{tr}(AYBX^T)$,
where $A$ is $m \times m$, $B$ is symmetric $n \times n$, and $X$ and $Y$ are $m \times n$ left stochastic matrices. I'm specifically interested when this occurs for $X = Y$. I attempted using a Lagrangian with the constraints $XX^T - J = 0$ or $X^T1_m - 1_n = 0$.
$\mathcal{L} = \text{tr}(AYBX^T) - \text{tr}(\Lambda^T (XX^T - J)) \implies \partial_X \mathcal{L} = AYB - (\Lambda + \Lambda^T)X = 0$,
$\mathcal{L} = \text{tr}(AYBX^T) - \lambda^T(X^T1_m - 1_n) \implies \partial_X \mathcal{L} = AYB - 1_m \lambda^T = 0$,
but those don't seem to work (I know this because I know the answers to some examples). I'm probably doing something wrong that's quite basic, so my apologies. Does anyone have any hints or recommendations?
Thank you very much.
Let $C=AYB$ and consider the function $$ \phi = {\rm tr}(CX^T) = C:X $$ where a colon denotes the Frobenius product, a convenient notation for the trace. It can also be written as elementwise/Hadamard multiplication followed by summation over all terms. $$\phi = \sum_{ij}(C\odot X)_{ij}$$
To minimize $\phi$ find the smallest element in each column of $C$ and set the corresponding element of $X$ equal to one. Set all other elements of $X$ to zero.
To maximize $\phi$, locate the largest element in each column of $C$ and set the corresponding element of $X$ equal to one.
The described $X$ matrices are obviously stochastic and maximize (or minimize) the given objective function.