I noticed somewhere, a function similar to $\|A-BCA\|_F^2$ was claimed to be non-convex regarding $B$ or $C$ individually, while a relaxation like $D=CA$ makes $\|A-BD\|_F^2$ convex respect to each $B$ and $D$ individually. The matrices $A, B, C, D$ are not square-matrix, and by individually, I mean assuming the value of one matrix fixed while analyzing w.r.t the other matrix.
Honestly, I do not understand why the first one is non-convex while the latter is convex?
Update:
In fact, i fund the above claim in This NIPS paper, where they claimed that Eq. 5 is not a convex problem w.r.t $P_k$, but the relaxation in Eq. 6 makes the objective convex w.r.t to each of the components.
Let us check that the first function is convex w.r.t. $C$. Thus, we have to show that the function $$t \mapsto \| A - B \, (C + t \, D) \, A\|_F$$ is convex for all matrices $D$. Let us abbreviate $G := A - B \, C \, A$ and $H := B \, D \, A$. Then, $$ \|A - B \, (C + t \, D) \, A\|_F = \|G - t \, H\|_F^2 = \|G\|_F^2 - 2 \, t \, (G,H)_F + t^2 \, \|H\|_F^2$$ and this function is clearly convex w.r.t. $t$. Here, $(\cdot,\cdot)_F$ is the Frobenius inner product.