Assume $A\in\mathbb{R}^{m\times n}$ is a $m$ by $n$ matrix. Let $|\cdot|$ denote the Frobenius norm of matrix. Define function $f:\mathbb{R}^{m\times n}\to\mathbb{R}$ as $f(A):=|A^TA|$. Is $f$ a convex function?
Intuitively, I think this function is a composition of a norm and something with quadratic structure, and thus should be convex.
To prove this, $f(tA+(1-t)B)=|t^2A^TA+t(1-t)(A^TB+B^TA)+(1-t)^2B^TB|\le t^2|A^TA|+(1-t)^2|B^TB|+t(1-t)|A^TB+B^TA|$.
If $|A^TB+B^TA|\le|A^TA|+|B^TB|$, then we get the convexity. Is this inequality true?
We have $$ |A^\top B|^2 = \operatorname{trace}(B^\top A A^\top B) = \operatorname{trace}(B B^\top A A^\top) = (B B^\top, A A^\top)_F \le |B B^\top| \, |A A^\top| = |B^\top B| \, |A^\top A|.$$
From this, we get $$|A^\top B + B^\top A| \le |A^\top B| + |B^\top A| \le 2 \, |B^\top B|^{1/2} \, |A^\top A|^{1/2} \le |A^\top A| + |B^\top B|.$$