I am currently confused about the notion of generalized Fourier transforms on discrete groups (I'm only concerned about discrete groups here, not necessarily abelian). On finite discrete groups, I understand that, given a function $f\colon G \to \mathbb{C}$, one can define generalized Fourier coefficients (which can be matrix coefficients) $f^*_\rho$ as $$f^*_\rho = \sum_g f(g)\rho(g)$$ where $\rho$ is an irreducible representation of $G$. This gives the usual discrete Fourier transform when $G=\mathbb{Z}/n\mathbb{Z}$, and there also have been applications for the analysis of ranked data when $G=S_n$ (the symmetric group on $n$ elements).
It seems to me that things get a little more complicated when one tries to do the same with infinite discrete groups. For example, if $G=\mathbb{Z}$, the irreducible representations of $G$ are the complex numbers, and this would give the Laplace transform, not the discrete-time Fourier transform. Is this because the group is infinite, and that the above formula cannot be used anymore ?
Also, I have read that one has to consider unitary representations, which in the case $G=\mathbb{Z}$ would amount to consider elements of the circle group (I suspect that, from a physical point of view, these are related to finite-energy basis functions). I'm a bit lost here, so any help to clarify these notions would be appreciated.
First, yes, you should consider unitary representations. The motivating example should be the group $ \mathbb{R} $ (you could also take $ \mathbb{T} $). There the Fourier transform is defined as
$ \hat f(\xi) = \int_{\mathbb{R}} f(x) \overline{e^{2 \pi i \xi x} } dx $
There are two concepts hidden in this transform. The functions $ \chi_{\xi}(x) = e^{2 \pi i \xi x} $ are nothing but unitary representations of the group $ \chi_{\xi} \colon \mathbb{R} \to \mathcal{U}(\mathbb{C}) \cong \mathbb{S}^1 $. Also, the Lebesgue measure $ \lambda $ is invariant under the action of the group:
$ \lambda( x + B ) = \lambda (B), \forall x \in \mathbb{R}, \forall B \subset \mathbb{R} $ measurable
This is also the way one tries to generalize this to locally compact groups (where such invariant Haar measures exist). This works completely fine for Abelian groups, where unitary representations are one dimensional, and the Fourier transform provides you a function $ \hat f $, which pointwise gives you a number.
It also works for compact groups. But it is a bit more involved, your transform gives you a map, which pointwise gives you an operator on a Hilbert space, so if $ \pi \colon G \to \mathcal{U}(H) $ is a unitary representation of your group $ G $ and $ f \in L^2(G, m_G) $ (where $ m_G $ is the corresponding Haar measure), your Fourier transform gets
$ \hat f (\pi) = \int_G f(x) \pi(x)^* dm_G(x) $
which is an operator on your Hilbert space $ H $ (defined in a weak sense using the inner product).
For non-Abelian, non-compact groups, it is difficult to say something generally. You want at least the Plancherel theorem to hold, i.e. the transformation should be an isometry. This can for example be done for unimodular, type I groups. Discrete groups are unimodular and they are type I iff they posses an Abelian, normal subgroup of finite index.
The Plancherel formula may hold under even weaker conditions on your group, but I'm not aware with all the different situations. E.g. there are non-unimodular groups, for which the Plancherel formula does not hold. The few things I know are from the (very recommendable) book "A course in Abstract Harmonic Analysis", G.B. Folland. Chapter 7 gives an overview of the non-Abelian, non-compact situation, Theorems 7.44, 7.50 state the Plancherel theorem for quite general cases.