If you write $G = \{g_1, g_2, . . . , g_n\}$, then you can identify a function on G by a vector with $n$ components. The j-th component is f(gj ), for each $j = 1, 2, . . . , n.$ Addition of functions is just vector addition, and similarly for scalar multiplication. Forgetting about scalar multiplication, this shows $L^2(G)$ is an abelian group under $+$. For $f, g \in L^2(G)$, define the convolution of $f$ and $g$, denoted $f ∗ g$ by $(f * g)(x) = \sum_{a,b \in G} f(a)g(b)$ for $x \in G$ with $ab = x$.
I need to show that $f ∗ g \in L^2(G)$, but I'm having a hard time understanding the notation of convolution in this definition.
To get your head around this, I think you'll need an example. Actually calculating this takes some time, but hopefully you'll appreciate the definition of $f * g$, and how it's a function.
For our example, consider the group $G = S_3$ (whose elements I'll notate in cycle notation). Let us define $f$ and $g$ by:
\begin{align*} f(e) &= 1 \\ f(\pmatrix{1&2}) &= -i \\ f(\pmatrix{1&3}) &= 1+i \\ f(\pmatrix{2&3}) &= 1 \\ f(\pmatrix{1&2&3}) &= 2-i \\ f(\pmatrix{1&3&2}) &= 1+6i \\ g(e) &= 0 \\ g(\pmatrix{1&2}) &= 2 \\ g(\pmatrix{1&3}) &= -i-1 \\ g(\pmatrix{2&3}) &= 1+3i \\ g(\pmatrix{1&2&3}) &= 0 \\ g(\pmatrix{1&3&2}) &= -3. \end{align*} These are just two functions in $L^2(G)$. Note that there's no restriction on what values we can assign to each group element, so long as they're all complex (even though I opted for Gaussian integers, to simplify calculation.
Let's compute $f * g$ at a couple of values. What about $(f * g)(e)$? We need to find all ordered pairs $a, b$ in $S_3$ whose product is $e$. That is, all the inverse pairs. Note that all the transpositions are self-inverse, as is $e$. The cycles are inverse to each other. This means that: \begin{align*} (f * g)(e) &= f(e)g(e) + f(\pmatrix{1 & 2})g(\pmatrix{1 & 2}) + f(\pmatrix{2 & 3})g(\pmatrix{2 & 3}) + f(\pmatrix{1 & 3})g(\pmatrix{1 & 3}) \\ &+ f(\pmatrix{1 & 2 & 3})g(\pmatrix{1 & 3 & 2}) + f(\pmatrix{1 & 3 & 2})g(\pmatrix{1 & 2 & 3}) \\ &= 1 \cdot 0 + (-i) \cdot 2 + 1 \cdot (1 + 3i) + (1 + i) \cdot (-i - 1) + (2 - i) \cdot (-3) + (1 + 6i) \cdot 0 \\ &= 0 - 2i + 1 + 3i - 2i - 6 + 3i + 0 \\ &= -5 + 2i. \end{align*}
That's just the value of the convolution at one point! We can also find, for example, $(f * g)(\pmatrix{1 & 2})$. We therefore need to find all the pairs $(a, b)$ of group elements that multiply to $ab = \pmatrix{1 & 2}$. These are, \begin{align*} (a, b) = \; &(e, \pmatrix{1 & 2}), (\pmatrix{1 & 2}, e), (\pmatrix{1 & 3}, \pmatrix{1 & 2 & 3}), (\pmatrix{2 & 3}, \pmatrix{1 & 3 & 2}), \\ &(\pmatrix{1 & 2 & 3}, \pmatrix{2 & 3}), (\pmatrix{1 & 3 & 2}, \pmatrix{1 & 3}). \end{align*} Hopefully you can verify that each pair $(a, b)$ has the product $\pmatrix{1 & 2}$. Then, the convolution is given by \begin{align*} (f * g)(\pmatrix{1 & 2}) &= f(e)g(\pmatrix{1 & 2}) + f(\pmatrix{1 & 2})g(e) + f(\pmatrix{1 & 3})g(\pmatrix{1 & 2 & 3}) \\ &+ f(\pmatrix{2 & 3})g(\pmatrix{1 & 3 & 2}) + f(\pmatrix{1 & 2 & 3})g(\pmatrix{2 & 3}) + f(\pmatrix{1 & 3 & 2})g(\pmatrix{1 & 3}) \\ &= 1 \cdot 2 + (-i) \cdot 0 + (1 + i) \cdot 0 + 1 \cdot (-3) + (2 - i) \cdot (1 + 3i) + (1 + 6i) \cdot (-i - 1) \\ &= 2 + 0 + 0 - 3 + 5 + 5i + 5 - 7i \\ &= 9 - 2i. \end{align*} As you can see, the scalar you get depends on the group element $x$ you feed it. And, hopefully, you can see how you'd continue this procedure for other $x \in S_3$ (or even, how you would do this for more general finite groups).