This is exercise 3.32 in Fulton and Harris' Representation Theory: A First Course.
It defines Fourier transform in a form unfamiliar to me, and I could not find any definition of Fourier transform like this after searching on the Internet. How does Fulton and Harris' definition relate to the familiar definition of Fourier series or Fourier transform (sources: Wikipedia entry for Fourier series, Wikipedia entry for Fourier transform) as following?
Fourier series:
$$ s(x) \sim A_0 + \sum_{n=1}^\infty \left(A_n \cos\left(\frac{2\pi nx}{P} \right) + B_n \sin\left(\frac{2\pi nx}{P}\right)\right) $$
Fourier transform:
$$ \hat{f}(\xi) = \int_{-\infty}^{\infty} f(x)\ e^{-i 2\pi \xi x}\,dx $$
For additional reference, the definition of a representation can also be found in Wikipedia.
This is a long story. This is the Fourier transform on a finite group. It's a generalization of the discrete Fourier transform, which it specializes to when $G = \mathbb{Z}/n\mathbb{Z}$. The sense in which the discrete Fourier transform deserves the name "Fourier transform" (aside from satisfying an inversion formula, a Plancherel formula, etc.) is that it is a special case of the Fourier transform on locally compact abelian groups, which relate functions on such a group $A$ to functions on its Pontryagin dual $\widehat{A}$, which can be identified with its irreducible representations. This version of the Fourier transform specializes to Fourier series when $A = S^1$ and to the usual Fourier transform when $A = \mathbb{R}$.
So, this is a generalization of a special case of a generalization of the usual Fourier transform, to a not-necessarily-abelian but finite group. It has a generalization to compact groups via the Peter-Weyl theorem, although I think most people wouldn't call that a Fourier transform. The case of nonabelian and noncompact groups is very complicated and I don't know anything about it, but it's studied in harmonic analysis.
For the finite group case you can see, for example, Terras' Fourier Analysis on Finite Groups and Applications.