Noncommutative dual group

Question

If $G$ is a locally compact group, we can define its dual group $\hat G$. That is set of continuous homomorphism from $G$ to circle group $\mathbb T$. My question is how to define dual group $\hat G$ when $G$ is a noncommutative group?

Answer 1

As mentioned in the comments, the "correct" generalization of the dual group to the noncommutative case is the unitary representation theory of $G$ (studying this reduces to studying homomorphisms into the circle group, or $\text{U}(1)$, when $G$ is commutative). At least when $G$ is compact, it is possible to recover $G$ from its unitary representation theory using some version of Tannaka reconstruction, e.g. the Doplicher-Roberts theorem; this is the "correct" generalization of Pontrjagin duality to this case.

There are other possible generalizations; see this MathOverflow question for some.