This is the problem I would like to solve:
Problem. Given $A\in\mathbb{R}^{N\times N}$, solve $$ \min_{T\in O(N)} \left\| T^\top A T\right\|_0, $$ where $\left\| X\right\|_0:=\left|\{(i,j)\,:\, [A]_{ij}\neq 0\}\right|$ and $O(N)$ is the group of $N\times N$ orthogonal matrices.
In less formal terms, I would like to find an "optimal" orthogonal transformation $\hat{T}$ that renders $A$ as sparse as possible.
Hence, my questions:
Has this problem been addressed in some works?
If not, do you have suggestions on how to solve the problem (or an approximation of it)?