Let $A,B$ be two $n\times n$ matrices with all elements being non-negative, i.e. $A(i,j)\ge 0,B(i,j) \ge 0$, the question is to check if optimization problem
$\min_{P,Q\in\Bbb R^{n\times n},P(i,j)\ge0,Q(i,j)\ge0}\|P^TAQ-B\|_F$
is convex or not, where all elements of $P,Q$ are non-negative, and $\|\|_F$ is Frobenius norm. I need help solving this problem and I want to ask what are the methods to determine non-convexity. Thanks a lot!
My attempt to find a counterexample for the Jensen's inequality is not successful and it looks very intimidating
$\begin{gathered} \left\| {(\theta P_1^T + (1 - \theta )P_2^T)A(\theta {Q_1} + (1 - \theta ){Q_2}) - B} \right\| = \left\| {{\theta ^2}P_1^TA{Q_1} + {{(1 - \theta )}^2}P_2^TA{Q_2} + \theta (1 - \theta )P_1^TA{Q_2} + \theta (1 - \theta )P_2^TA{Q_1}} \right\| = {\theta ^2}\left\| {P_1^TA{Q_1} - B} \right\| + {(1 - \theta )^2}\left\| {P_2^TA{Q_2} - B} \right\| + \theta (1 - \theta )\left\| {P_1^TA{Q_2}} \right\| + \theta (1 - \theta )\left\| {P_2^TA{Q_1}} \right\| \hfill \\ \leqslant ?\theta \left\| {P_1^TA{Q_1} - B} \right\| + (1 - \theta )\left\| {P_2^TA{Q_2} - B} \right\| = \left\| {\theta P_2^TA{Q_2} + (1 - \theta )P_2^TA{Q_2} - B} \right\| \hfill \\ \end{gathered} $
Currently I guess this optimization is non-convex.
Very often the easiest approach is by explicit counterexample. In your case, look at the scalar case, let $A=B=1$ and study the values $(P,Q)=(2,1)$ and $(1,2)$ vs the middle $(1.5,1.5)$.