Suppose $G$ is a compact group and $\rho$ is an irreducible unitary representation of $G$ (in particular, its finite-dimensional). I am interested whether one can say something about the multiplicity of $\rho$ in its symmetric square $\mathrm{Sym}^2(\rho)$ (assuming that its contained). For some examples, I can compute the character inner product $$ \langle \chi_\rho , \chi_{\mathrm{Sym}^2(\rho)} \rangle, $$ where $\chi_{\mathrm{Sym}^2(\rho)}(g) = (\chi_\rho(g)^2 + \chi_\rho(g^2))/2$, but I lack some general understanding.
I can make the following additional assumptions on $(G,\rho)$:
- $G\subset\mathrm{U}(d)$ is a subgroup of the unitary group and $\rho$ is an irrep of the representation $U \mapsto U(\cdot)U^{-1}$ acting on complex $d\times d$ matrices. In particular, $\rho$ is a real representation.
- $d = 2^n$ where $n > 1$
- If this simplifies matters, we may assume that $G$ is finite.
Is there any chance of computing the character inner product or some other way of at least gaining a partial understanding of the possible multiplicities? I am grateful for any hints.
Frequently in practice this multiplicity will just be zero, for the following reason: often $G$ will have a nontrivial center $Z(G)$. By Schur's lemma, the center acts by scalars on any irreducible representation, so any irreducible representation has a central character which is given by some $1$-dimensional character $\lambda : Z(G) \to \mathbb{C}^{\times}$ of the center. If $V$ has central character $\lambda$ then $V^{\otimes 2}$ has central character $\lambda^2$ and so does any subrepresentation of it, hence in particular so does the symmetric square. And representations with different central character cannot map to each other. So if $\lambda \neq \lambda^2$, which equivalently says that $\lambda \neq 1$ is nontrivial, then every map $V \to S^2(V)$ is zero. This happens, for example, for $G = SU(n), V = \mathbb{C}^n$ where the center is $\mathbb{Z}/n$ and the central character is faithful.
So if the multiplicity is nonzero then the central character of $V$ must be trivial (e.g. this could happen if $G$ itself has trivial center). Other than that I'm not sure what can be said. By a dimension count the multiplicity is at most $\left\lfloor \frac{\dim V - 1}{2} \right\rfloor$ but I don't know a nontrivial example off the top of my head where it's even nonzero.