Invariant inner products on infite-dimensional representations

Question

Let $G$ be a compact group and let $V$ be it's continuous representation. It is well known that if $V$ is finite-dimensional, then there is an $G$-invariant inner product on $V$. I haven't found a corresponding statement in the literature for infinite-dimensional representation. What breaks down? Is there a simple counterexample?

Answer 1

Theorem: Let $G$ be a compact (Hausdorff) group and let $H$ be a Hilbert space on which $G$ acts strongly continuously. Then

$$\langle u, v \rangle_G = \int_G \langle gu, gv \rangle \, dg$$

defines a $G$-invariant inner product on $H$.

Proof. By assumption $g \mapsto \langle gu, gv \rangle$ is a continuous function on $G$, so it is integrable and the above integral exists. From here the only nontrivial step is showing positive-definiteness (everything else follows from linearity of integration and invariance of Haar measure). So suppose $v \in H$ is a nonzero vector. Let $U$ be an open neighborhood of the origin so that $\langle gv, gv \rangle \ge \epsilon$ for all $g \in U$ and for some $\epsilon > 0$. Since $G$ is compact, the set of translates $gU$, which cover $G$, admits a finite subcover $g_1 U, ... g_n U$, from which it follows that $U$ has positive measure (at least $\frac{1}{n}$) and that

$$\langle v, v \rangle_G \ge \frac{\epsilon}{n}.$$

The conclusion follows.