In some work I've been doing, I've come across a problem which involves a family of symmetric matrices $A_k\in\mathbb R^{n\times n}$ for $1\le k\le m$. The problem is very well-behaved when $A_k$ pair-wise commute, but things become harder if the $A_k$ are very far from commuting. What I would like to do, then, is to maximize $$ \| [A,B]\|_{HS}^2,\quad\hbox{ subject to } \|A\|_{HS},\|B\|_{HS}\le1 $$ where $\|\cdot\|_{HS}$ denotes the Hilbert-Schmidt norm, $\|A\|_{HS}^2=\sum_{i,j}A_{ij}^2=\operatorname{tr}(A^TA)$.
I have found that for any maximal pair $(A,B)$, we must have that $$ \operatorname{tr}ACBB-\operatorname{tr}CBAB=\lambda\operatorname{tr}CA, \quad \operatorname{tr}AACB-\operatorname{tr}ABCA=\eta\operatorname{tr}CB $$ for all $C\in\mathbb R^{n\times n}$, and $\lambda,\eta\ge0$ (this is equivalent to $(A,B)$ being a local maximum). This is equivalent to the condition that $$ ABB-BAB=\lambda A,\quad BAA-ABA=\eta B. $$ But can this condition be reduced further? I'm really more interested in finding the maximum, than finding which $A,B$ attain it.
Since $A$ is symmetric, we can choose a basis to make it diagonal. Reordering the basis if necessary, we can make the diagonal entries $\lambda_1, \ldots, \lambda_n$ of $A$ ascending (i.e. $\lambda_1 \le \cdots \le \lambda_n$).
Let $B = (b_{ij})$, we have $( AB - BA)_{ij} = b_{ij}(\lambda_i-\lambda_j)$. This leads to
$$\| [A, B] \|_{HS}^2 = \sum_{i=1}^n\sum_{j=1}^n b_{ij}^2 (\lambda_i - \lambda_j)^2 \le \sum_{i=1}^n\sum_{j=1}^n b_{ij}^2 (\lambda_n - \lambda_1)^2 = \|B\|^2_{HS}(\lambda_n - \lambda_1)^2 \le (\lambda_n - \lambda_1)^2 $$
Since $\lambda_1^2 + \lambda_2^2 + \cdots \lambda_n^2 = \|A\|^2_{HS} \le 1$, we have
$$(\lambda_n - \lambda_1)^2 \le (\lambda_n - \lambda_1)^2 + (\lambda_n + \lambda_1)^2 = 2(\lambda_n^2 + \lambda_1^2) \le 2\tag{*1}$$
From this, we can conclude
$$\|[A,B]\|_{HS} \le 2\quad\text{ for symmetric }\;\; A, B \;\;\text{ whenever }\;\; \| A \|_{HS}, \| B \|_{HS} \le 1\tag{*2}$$
Notice in $(*1)$, the equality is achieved when $$\lambda_1 = -\frac{1}{\sqrt{2}}, \lambda_2 = \cdots = \lambda_{n-1} = 0, \lambda_n = \frac{1}{\sqrt{2}}$$
With this as a hint, we find
$$A = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 0 & 0 & \cdots\\ 0 &-1 & 0 & \cdots\\ 0 & 0 & 0 & \cdots\\ \vdots & \vdots & \vdots & \ddots \end{bmatrix} \quad\text{ and }\quad B = \frac{1}{\sqrt{2}}\begin{bmatrix} 0 & 1 & 0 & \cdots\\ 1 & 0 & 0 & \cdots\\ 0 & 0 & 0 & \cdots\\ \vdots & \vdots & \vdots & \ddots \end{bmatrix} $$ is an example where the upper bound $2$ in $(*2)$ is achieved.