Frobenius and operator $2$-norm

Question

I have been studying about norms and for a given matrix $A$, I haven't been able to understand the difference between Frobenius norm $\|A\|_F$ and operator $2$-norm $\|A\|_2$. Can someone help me understand the difference between them?

Answer 1

$\|A\|_F$ is $\|\operatorname{vec}(A)\|_2$, the Euclidean norm of the vector $\operatorname{vec}(A)$ obtained by stacking the columns of $A$ one above the other. So, you just reshape $A$ into a vector and take its Euclidean norm. The Frobenius norm of $A$ can also be expressed as $\sqrt{\operatorname{tr}(A^\ast A)}$, because each diagonal entry of $A^\ast A$ is the squared Euclidean norm of a row of $A$.

$\|A\|_2$, in contrast, is the maximum possible Euclidean norm of $Av$ for a unit vector $v$. Since $A$ a linear operator that operates on $v$, we call it an operator norm. And as we consider the $2$-norm of $Av$, the matrix norm $\|A\|_2$ is also called an induced norm. Had the vector $p$-norm been used in place of the $2$ norm, the resulting matrix norm $\|A\|_p=\max_{\|v\|_p=1}\|Av\|_p$ is called an operator/induced $p$-norm.

For example, $$ \|I_2\|_F=\left\|\pmatrix{1\\ 0\\ 0\\ 1}\right\|_2=\sqrt{2}\ \text{ but } \ \|I_2\|_2=\max_{\|v\|_2=1}\|I_2v\|_2=\max_{\|v\|_2=1}\|v\|_2=1. $$

Both the operator $2$-norm (but not other induced $p$-norms) and Frobenius norm are unitarily invariant, i.e. $\|UAV\|_F=\|A\|_F$ and $\|UAV\|_2=\|A\|_2$ whenever $U,V$ are unitary matrices. Therefore, by singular value decomposition, we always have $$ \|A\|_F = \sqrt{\sum_i\sigma_i(A)^2}\ge\sigma_1(A)=\|A\|_2 $$ and the two matrix norms are equal only when $\sigma_2(A)=\cdots=\sigma_n(A)=0$, i.e. when $\operatorname{rank}(A)\le1$.

The operator $2$-norm is also confusingly called spectral norm in the literature, but as we have seen in the above, $\|A\|_2$ is the largest singular value of $A$. It is not really about the spectrum of $A$. The term probably originated from considering the spectrum of $A^\ast A$.