Trigonometric polynomials on non-compact and non-abelian groups

Question

Hewitt and Ross define trigonometric polynomial on a locally compact group $G$ as a linear combination of matrix elements of continuous unitary irreducible representations of $G$: $$ f(t)=\sum_{i=1}^n \lambda_i\cdot\langle\pi_i(t)x_i,y_i\rangle,\qquad t\in G, \quad \lambda_i\in{\mathbb C},\quad \pi_i:G\to B(H_i),\quad x_i,y_i\in H_i. $$ If $G$ is abelian or compact then the space ${\tt Trig}(G)$ of trigonometric polynomials on $G$ is an algebra with respect to the pointwise multiplication.

This is strange, I can't find mentionings of the same proposition in the non-abelian and non-compact case. Is it possible that in general case (for arbitrary locally compact group $G$) the space ${\tt Trig}(G)$ is not an algebra?

I would be grateful for advices on what one can read about this.

Answer 1

This question has been asked and answered on MathOverflow. I have replicated the accepted answer by Yemon Choi below.

It isn't an algebra: the tensor product of irreducibles does not always decompose as a finite sum of irreducibles, although in some of those cases it may decompose as a direct integral of irreducibles.

For instance, take $G=H_3({\bf R})$, the group of upper-triangular matrices with real entries and with $1$ on the diagonal. This has very nice representation theory from the point of view of harmonic analysis (it is not only Type I, but it is what Kaplansky called CCR, that is $\pi(L^1(G))\subseteq {\mathcal K}(H_\pi)$ for every continuous irreducible unitary representation $\pi: G \to {\mathcal U}(H_\pi)$). There is a continuous quotient homomorphism $q:G \to G/Z(G) \cong {\bf R}^2$.

Now for $t\neq 0$ there is an infinite irreducible unitary representation $\pi_t: G \to {\mathcal U}(L^2({\bf R}))$ (in some realizations called the Schrödinger representation). It is true that if $s$, $t$ and $s+t$ are all non-zero, and $f$ is a coefficient function of $\pi_s$ and $g$ is a coefficient function of $\pi_t$, then $fg$ is the norm-limit inside $B(G)$ of sums of coefficient functions of $\pi_{s+t}$. I don't know if $fg$ is actually a coefficient function itself.

However, the representation $\pi_t\otimes \pi_{-t}$ is equivalent to the representation $\lambda_{\bf R{^2}} \circ q: G \to {\mathcal U}(L^2({\bf R}^2))$ which is not (quasi-)equivalent to a direct sum of irreducibles. Consequently there should exist $f$ a coefficient function of $\pi_t$ and $g$ a coefficient function of $\pi_{-t}$ such that $fg$ does not belong to the closure of ${\rm Trig}(G)$ inside $B(G)$.

Similar example should exist for e.g. ${\rm SL}(2,{\bf R})$ but I don't remember the ``fusion rules'' for its irreducible representations off the top of my head.

On the positive side: if $G$ is a Moore group then the completion of ${\rm Trig}(G)$ inside $B(G)$ is indeed a subalgebra of $B(G)$. This may be folklore: I learned of it from the paper http://arxiv.org/abs/1208.1519 which has more information on "negative examples".