The problem I'm trying to solve is the following. Given an input tensor $A_{ij}^{\alpha\beta}$, I want to find the unitary tensor $U^{ij}_{kl}$ such that the function $f = -\sum_i s_i^2\log s_i^2$ is minimized, where the $s_i$ are the singular values of the vectorization $U^{ij}_{kl}A_{ij}^{\alpha\beta} = X^{\alpha\beta}_{kl}\rightarrow X_{(\alpha i)(\beta j)}$.
I've tried a few computational approaches involving parameterizing the matrix $U$ using a Cayley transform, then applying various optimization methods, but I haven't found something that consistently finds an optimal or near-optimal solution.
Is there a better way of solving this problem? Even if there is no direct analytic solution, is a more natural parameterization that would make the cost function surface smoother?