I have an optimization problem as follows: $$\min_{\substack{A\in \mathbb R^{k\times k}\\P\in \mathbb O^{n\times k}}}\|B - P \cdot A\cdot P^T \|_F^2$$ where $B$ (whose rank can be thought as greater than or equal to $k$) is given, $\mathbb O^{n\times k} (n> k)$ represents the set of matrices with orthonormal columns in $\mathbb R^n$, and I am interested in the optimal matrix $P$. This can be thought as finding a $k$-dimensional subspace that is close to $B$'s column and row space.
The problem is easy if $B$ is symmetric, which can be solved using eigenvalue decomposition (or SVD). But I am a bit lost if $B$ is not symmetric.
This problem is equivalent to $$\begin{aligned}&\max_{P} \quad \|P^TB P\|_F^2\\&s.t. \quad P\in\mathbb O^{n\times k}\end{aligned}$$
I can solve this problem for some special cases. For example, if $B$ is rank $k$, and it has a SVD as $B=\sum_{i=1}^{k-1}u_iu_i^T+\sigma_ku_kv_k^T$, where $\sigma_k<1$ so that $\sigma_k$ is the smallest singular value. In this case, $k-1$ row and column vectors are in the same space, and I will choose $\{u_i,i=1,\ldots,k-1\}$ as the first $k-1$ columns of $P$. For the last one, I should choose the bisector between vector $u_k$ and $v_k$. But I don't know how to deal with a more general case. For example, when there are only $k-2$ row and column vectors (with big singular values) in the same space, and I need to choose two more extra vectors in addition to these $k-2$ vectors.
Any ideas related to this problem would be appreciated. I think this should be a well-studied problem in mathematics, but I failed to search the answer for it.