I have unitary representations of an infinite group (but finitely generated) on an infinite dimensional Hilbert space. I would like to show that two representations are not equivalent. I am guessing I would need to find some invariant property of an representation and then show the two representations do not share the same invariant property. However after quite a bit of searching on the internet, I could not find any invariants that apply to representations on an infinite group. All I could find were invariants of representations of finite groups.
Are there are any invariants of representations of infinite groups? Are there any other methods of proving two representations are not equivalent? Also in general, I would like to learn more about representations of infinite groups. Could you recommend some resources I could look at?
To fix notation let $u_1$ and $u_2$ be two unitary representations of the group $G$ on Hilbert spaces $H_1$ and $H_2$.
One of the most elementary reasons why $u_1$ and $u_2$ might fail to be unitarily equivalent is spectra: if there is an element $g$ in $G$ such that the spectrum of $u_1(g)$ differs from the spectrum of $u_2(g)$, then clearly $u_1$ and $u_2$ are not unitarily equivalent.
This is not such a fine invariant, though, since representations if infinte groups often involve only unitary operators with full spectrum, but of course this is a start.
The representation theory of $G$ is equivalent to the representation theory of the Banach $^*$-algebra $\ell ^1(G)$, in the sense that each $^*$-representation of the latter restricts to a unitary representation of $G$ and, conversely, every representation $u$ of $G$ integrates to a representation $\pi $ of $\ell ^1(G)$ according to the formula $$ \pi (f) = \sum_{g\in G}f(g) u_g, \quad \forall f \in \ell ^1(G). $$
The advantage of working with $\ell ^1(G)$ is the availability of a linear structure. For example, the kernel of the integrated representation $\pi $ is a two-sided ideal in $\ell ^1(G)$, and is thus an invariant for the representation $u$ from which $\pi $ has been integrated.
The envelopping C$^*$-algebra of $\ell ^1(G)$, called $C^*(G)$, is also a very useful gadget because, like $\ell ^1(G)$, its representation theory is also equivalent to the representation theory of $G$. However it has several advantages due to the fact that its representation theory is highly developped.
To be a bit more precise, given a unitary representation $u$ of $G$, the integrated representation $\pi $ on $\ell ^1(G)$ extends to $C^*(G)$ due to the universal properties of the envelopping C$^*$-algebra, and if $\rho $ denotes this extension, then again the kernel of $\rho $ is an invariant of $u$.
Given the finer structure of $C^*(G)$, the kernel of $\rho $ carries much more information about $u$. For instance, if two unitary representations $u_1$ and $u_2$ give rise to representations $\rho _1$ and $\rho _2$ of $C^*(G)$ with the same kernel, then $u_1$ and $u_2$ are mutually weakly contained in each other in the sense that, for every $\xi $ in $H_1$, the positive type function on $G$ given by $$ g\in G\mapsto \langle u _1(g)\xi ,\xi \rangle \in {\mathbb C} $$ can be approximated by positive type functions coming from vectors in the representation space of $u_2$, and vice-versa.
Another very useful algebraic invariant is the commutant of a unitary representation $u$, defined to be the set $u(G)'$ formed by all bounded, linear operators $T$ commuting with every $u_g$.
$u(G)'$ is a von Neumann algebra and it carries a lot of information about multiplicities of sub-representations of $u$. For instance, $u(G)'$ is isomorphic to ${\mathbb C}$ iff $u$ is irreducible while $u(G)'$ is abelian iff $u$ is multiplicity free.
The von Neumann algebra generated by $u(G)$, namely the commutant of $u(G)'$, is a further very important invariant. Given the rich theory of classification of von Neumann algebras in types, many invariants can be extracted from $u(G)''$.