For any locally compact Hausdorff abelian group (LCA group) $A$, a character $\xi\colon A\mapsto\mathbb{T} $ is definined as a continuous group homomorphism to the unit circle $\mathbb{T}\subseteq\mathbb{C}$. The space of characters is called the dual space of $A$ and is denoted $\hat{A}$. To my knowledge, characters are vital for Fourier analysis on abelian groups, e.g. establishing the homeomorphism $\hat{A}\rightarrow \mathrm{sp}L^1(A),\,\xi\mapsto m_\xi$ (where $\mathrm{sp}$ denotes the Gelfand spectrum and $m_\xi(f):=\hat{f}(\xi)=\int_A f(x)\,\overline{\xi(x)}\,\mathrm{d}x$) and the Pontryagin duality $A\rightarrow\hat{\hat{A}},\,x\mapsto\delta_x$ (where $\delta_x(\xi):=\xi(x)$).
For non abelian locally compact Hausdorff abelian group it seems to be more complicated, as representations (which I do not understand yet) are being used instead of characters. Are there clear examples why characters fail in the non abelian case?
The short answer is that if $G$ is a group and $A$ an abelian group (like $\mathbb T$), then every group homomorphism $\phi\colon G\to A$ factors through the abelianization $G/[G,G]$. Here $[G,G]$ is the subgroup generated by commutators, i.e. elements of the form $ghg^{-1}h^{-1}$ with $g,h\in G$. Thus continuous group homomorphisms from $G$ to $\mathbb T$ can really only tell you things about the abelianization, which is not enough to understand the group itself. As an example, the free group $\mathbb F_d$ and the free abelian group $\mathbb Z^d$ have the same abelianization and thus the same group homomorphisms to $\mathbb T$.