Let $\Pi (G)$ be the set of equivalence classes of irreducible representations of a finite group $G$. Here $d_{\pi}$ denotes the dimension of the representation $\pi\in\Pi(G)$, and $\pi_{i,j}(x)$ denotes the (i,j) matrix entry of $\pi(x)$.
We define $\pi(f)=\sum_{x\in G}f(x)\pi(x)$ and we know that \begin{equation} \tag{1} f(1)=\frac{1}{|G|}\sum_{\pi\in\Pi(G)}d_{\pi} \operatorname{Tr}\left(\pi(f)\right) \end{equation} and \begin{equation} \tag{2} \sum_{x\in G}|f(x)|^2=\frac{1}{|G|}\sum_{\pi\in\Pi(G)}d_{\pi}\|\pi(f)\|^2_{HS}, \end{equation} where $\|.\|$ denotes the Hilbert-Schmidt norm (the square root of the sum of the absolute value squared of the entries). I try to show that (1) and (2) are equivalent.
Attempt: First I show (1) $\Rightarrow$ (2). We define the convolution on $G$ by $f*h(x)=\sum_{y\in G}f(xy^{-1})h(y)$. We apply the function $f*f^*$ to the equation (1), where $f^*(x)=\overline{f(x^{-1})}$ Then we get $$f*f^*(1)=\frac{1}{|G|}\sum_{\pi\in\Pi(G)}d_{\pi} \operatorname{Tr}\left(\pi(f*f^*)\right)$$ By the definition of convolution we have $f*f^*(1)=\|f\|_2^2$. Therefore it is enough to show that $\operatorname{Tr}(\pi(f*f^*))=\|\pi(f)\|^2_{HS}$. I know that \begin{align*} \operatorname{Tr}(\pi(f*f^*))&=\operatorname{Tr}(\pi(f)\pi(f^*))=\operatorname{Tr}(\pi(f))\operatorname{Tr}(\pi(f^*))\\&=\left(\sum_{i=1}^{d_\pi}\sum_{x\in G}f(x)\pi_{i,i}(x)\right)\left(\sum_{j=1}^{d_\pi}\sum_{y\in G}f^*(y)\pi_{j,j}(y)\right) \\&= \sum_{1\leq i,j,\leq d_{\pi}}\left(\sum_{x\in G}f(x)\pi_{i,i}(x)\right)\left(\sum_{y\in G}\overline{f(y^{-1})}\pi_{j,j}(y)\right) \end{align*} On the other hand,
\begin{align*} \|\pi(f)\|^2_{HS} &=\sum_{1\leq i,j,\leq d_{\pi}}\Big|\sum_{x\in G}f(x)\pi_{i,j}(x)\Big|^2=\sum_{1\leq i,j,\leq d_{\pi}}\sum_{x\in G}f(x)\pi_{i,j}(x)\overline{\sum_{y\in G}f(y)\pi_{i,j}(y)} \end{align*}
But I didn't find how to show the equality $$\sum_{1\leq i,j\leq d_{\pi}}\sum_{x\in G}f(x)\pi_{i,i}(x)\sum_{y\in G}\overline{f(y^{-1})}\pi_{j,j}(y)=\sum_{1\leq i,j\leq d_{\pi}}\sum_{x\in G}f(x)\pi_{i,j}(x)\overline{\sum_{y\in G}f(y)\pi_{i,j}(y)}$$
You use $\mathrm{tr}(AB)=\mathrm{tr}(A)\mathrm{tr}(B)$, but this isn't true. (What is true is $\mathrm{tr}(A\otimes B)=\mathrm{tr}(A)\mathrm{tr}(B)$, but we're not talking about tensor products.)
Instead, recognize $\pi(f^{\ast})=\pi(f)^{\ast}$ where $A^{\ast}$ denotes the conjugate transpose (I'm assuming your irreps are unitary?), and use the formula for the norm $\|A\|_{HS}^2=\mathrm{tr}(AA^{\ast})$ for operators $A$.
Nice idea using convolution to do $(1)\Rightarrow(2)$.