My major is physics. I need to use some tools in group theory, but I am really confused by the trace in compact infinite groups. The following is my question:
In discrete group theory, the irreducible representation of identity group element $e$ is always an identity matrix. So the trace of $e$ under regular representation is the order of the group : $\chi(e) = |G|$. I hope to get similar result for Lie group. For example, SO(2) has infinite number of irreducible representations, $D^{(m)}$, where $m=0,\pm 1, \pm 2,\cdots $. All of them are 1-dimensional $D^{(m)} = e^{im\phi}$. Here $\phi$ is the rotation angle. For element $R(\phi)$ in SO(2), the trace in $m$th irreducible representation should be $\chi(R(\phi))=e^{im\phi}$, if we sum all these traces up, we get the trace in regular representation. So what is the trace of $e$ in the regular representation ? It seems to be infinite.
As you implicitly observe, there is no well defined notion of the trace of this particular infinite-dimensional matrix. (There is a class of (infinite-dimensional) matrices to which one can assign traces, inventively called trace class (https://en.wikipedia.org/wiki/Trace_class), but there are no infinite-dimensional unitary matrices among them.) Notice that the statement that there is no trace is different to the statement that the trace is infinite. For example, the regular representation of $S^1$ on itself may be diagonalised so that the image of $z$ is the matrix $(z^n[n = m])_{m, n \in \mathbb Z}$, and there's no reasonable answer for the sum $\sum_{n \in \mathbb Z} z^n$ that doesn't make some assumptions that will destroy the nice invariance properties that we'd like a trace to have.
What one may do instead is to remember that the scalar character is just the trace of the distribution character, in the sense that $\operatorname{tr} \int \pi(g)f(g)\mathrm dg = \int \Theta_\pi(g)f(g)\mathrm dg$ for $f \in C^\infty(G)$ and $\pi$ an irreducible (finite-dimensional) representation of the compact Lie group $G$, where $\mathrm dg$ is a Haar measure on $G$; and one may ask about the distribution character of the regular representation. This is the $\delta$ distribution at the identity, in the sense that $$\sum_{\pi \in \hat G} \operatorname{deg}(\pi)\operatorname{tr} \int \pi(g)f(g)\mathrm dg = f(e)$$ (if $\mathrm dg$ is normalised to give $G$ total mass $1$). This is known as the Plancherel theorem.
EDIT: The original version of my second paragraph directly contradicted the first, by (inadvertently) not really speaking of the distribution character at all. The reason that I prefer to work via this roundabout approach of looking for a scalar function representing the distribution character $f \mapsto \int \pi(g)f(g)\mathrm dg$, rather than just taking the trace of $\pi(g)$ as one is used to doing, is precisely because it handes the infinite-dimensional case with élan. This is particularly important in the theory of non-compact groups, which may not even have any finite-dimensional representations.