Is there some sort of irrep orthogonality for finite groups?

Question

Disclaimer: As pointed out by several people the original orthogonality condition I was asking about seems to be incorrect so I'm just going to change the question to the orthogonality condition addressed in the answer below.

---

Let $ \pi_1, \pi_2 $ be distinct irreps of a finite group $ G $. Let $ \chi_1, \chi_2 $ be the corresponding characters. Then we know that
$$ \sum_{g \in G} \chi_1(g) \chi_2(g)^*=0 $$ Dropping the trace (and assuming that $ \pi_1, \pi_2 $ have the same degree in particular $ \pi_1: G \to GL_d(\mathbb{C}) $ , $ \pi_2: G \to GL_d(\mathbb{C}) $ ) consider the sum $$ \sum_{g \in G} \pi_1(g) \pi_2(g)^* $$ Is there some sort of "orthogonality" here as well? In other words, for two distinct irreps $ \pi_1,\pi_2 $ of the same degree do we have that $$ \sum_{g \in G} \pi_1(g) \pi_2(g)^* $$ is the zero matrix?

Of course this is true for degree $ 1 $ by character orthogonality. Is it true for higher degree irreps?

Answer 1

Let's generalize. Suppose $(V_1,\pi_1)$ and $(V_2,\pi_2)$ are two irreps of $G$. Define

$$ T(M)=\frac{1}{|G|}\sum_{g\in G} \pi_1(g)M\pi_2(g^{-1}). \tag{$\color{green}{\triangle1}$}$$

Observe $T$ is a linear operator on the vector space $\hom(V_2,V_1)$. Your original question was when $\dim V_1=\dim V_2$ and $M=I$, but this restriction is not necessary.

Observe that for any $M\in\hom(V_2,V_1)$, the operator $T(M)$ is equivariant, i.e. for any $a\in G$,

$$ \pi_1(a)T(M)=T(M)\pi_2(a), $$

by reindexing the summation and using socks-and-shoes. By Schur's lemma, if $\pi_1\not\cong\pi_2$ there are no intertwiners, so $T(M)=0$ for all $M$.

If $\pi_1=\pi_2$ then $T(M)=\lambda(M)I$ for some linear scalar function $\lambda(M)$ of $M$. Taking the trace in the definition $({\color{green}\triangle})$ yields $\mathrm{tr}\,M=d_1\lambda(M)$, where $d_1$ is the dimension, so

$$ T(M)=\delta_{\pi_1\pi_2}\frac{\mathrm{tr}\,M}{d_1}I. \tag{$\color{green}{\triangle2}$}$$

Note this implies the worry about coordinate-dependence expressed in the comments is not a problem. Indeed, if we define the conjugate $\pi_1'(g)=A\pi_1(g)A^{-1}$ for some $A\in\mathrm{GL}(V_1)$, and define $T'$ accordingly, we get $T'(M)=AT(A^{-1}M)$, so the same conclusions holds for $T'$ as $T$, and similarly if we use a conjugate of $\pi_2$ as well (assuming $\pi_1\not\cong\pi_2$).

---

Recall we have an isomorphism of reps $\hom(V_2,V_1)\cong V_1\otimes V^\ast_2$. Say we view $V_1,V_2$ as spaces of column vectors. We can view $V_2^\ast$ either as the same vector space and apply $\pi_2^\ast(g)=\pi_2(g^{-1})^T$ from the left, or we can view it as a space of row vectors on which we apply $\pi_2(g^{-1})$ from the right. This allows us to reinterpret $T$ from the linear operator $(\color{green}{\triangle1})$ on $\hom(V_2,V_1)$ to a linear operator $({\color{Magenta}\diamondsuit})$ on $V_1\otimes V_2^\ast$ ("transport of structure") given by

$$ T=\frac{1}{|G|}\sum_{g\in G} \pi_1(g)\otimes \pi_2(g^{-1})^T, \tag{$\color{Magenta}\diamondsuit$}$$

where we interpret $V_2^\ast$ as comprised of column vectors. If we view $V_2^\ast$ instead as comprised of row vectors, then the isomorphism $V_1\otimes V_2^\ast\cong\hom(V_2,V_1)$ is given by $v_1\otimes v_2^T\mapsto v_1v_2^T$ and we don't transpose $\pi_2(g^{-1})$ anymore; this is why $\pi(g^{-1})$ does not appear with a transpose in $({\color{green}\triangle})$.

---

We can generalize further still to include your previous question:

$$ T(M)=\frac{d_3}{|G|}\sum_{g\in G}\chi_3(g^{-1})\,\pi_1(g)M\pi_2(g^{-1}), \tag{$\color{blue}☾$}$$

where a third irrep $(V_3,\pi_3)$ has character $\chi_3=\mathrm{tr}\,\pi_3$ and dimension $d_3=\chi_3(e)$. Now, $T$ is the isotypical projector onto the $V_3$-component of $\hom(V_2,V_1)$ as a rep. For $\pi_3$ trivial this re-establishes the above conclusions.

---

One textbook application of $T(M)=\delta_{\pi_1\pi_2}\frac{\mathrm{tr}\,M}{d_1}I$ from $({\color{green}\triangle})$ is Schur Orthogonality for Matrix Coefficients. This was relevant to your next question.

If we pick the standard basis matrix $M=e_{jk}$ and look at the $i\ell$ entry of $T(M)$, on the one hand we get $\delta_{\pi_1\pi_2}\frac{\mathrm{tr}\,e_{jk}}{d_1}I_{i\ell}=\delta_{\pi_1\pi_2}\delta_{jk}\delta_{i\ell}/d_1$ from ($\color{green}{\triangle2}$). On the other hand, from ($\color{green}{\triangle1}$) we get

$$ \frac{1}{|G|}\sum_{g\in G}\sum_{j',k'}\pi_1(g)_{ij'}(\delta_{jj'}\delta_{kk'})\pi_2(g^{-1})_{k\ell}. $$

Assuming $\pi_2$ is unitary so $\pi_2(g^{-1})=\pi_2(g)^\dagger=\overline{\pi_2(g)}^T$, this simplifies to

$$ \frac{\delta_{\pi_1\pi_2}\delta_{i\ell}\delta_{jk}}{d_1} = \frac{1}{|G|}\sum_{g\in G}\pi_1(g)_{ij}\overline{\pi_2(g)}_{\ell k}. \tag{$\color{Purple}{\square}$}$$

These are the orthogonality relations.

---

If we (wlog) unitarize our irreps, then $\pi_2(g^{-1})^T=\overline{\pi_2(g)}$. Then we may set

$$ S(M)=\frac{1}{|G|}\sum_{g\in G}\pi_1(g)M\overline{\pi_2(g)}. \tag{$\color{orange}{\circ1}$}$$

The $i\ell$ matrix entry of $S(M)$ is given by

$$ S(M)_{i\ell} \,=\, \frac{1}{|G|}\sum_{g\in G}\sum_{j,k}\pi_1(g)_{ij}M_{jk}\,\overline{\pi_2(g)}_{k\ell} $$

$$ =\, \sum_{j,k}M_{jk}\big(\frac{1}{|G|}\sum_{g\in G} \pi_1(g)_{ij}\overline{\pi_2(g)}_{k\ell}\big) $$

$$ =\, \sum_{j,k}M_{jk}\frac{\delta_{\pi_1\pi_2}\delta_{ik}\delta_{j\ell}}{d_1} \,=\, \frac{\delta_{\pi_1\pi_2}}{d_1} M_{\ell i}. $$

Therefore we may conclude

$$ S(M)=\frac{\delta_{\pi_1\pi_2}}{d_1}M^T. \tag{$\color{orange}{\circ2}$}$$

Answer 2

What you wrote isn't right, and my guess is that basically any example will break it. It can be fixed though by adding a transpose:

$$ \sum_{g \in G} \pi_1(g) \pi_2(g^{-1})^\top = 0$$

Let's go back to character orthogonality to see where this is coming from. I claim that sum $\sum_{g \in G} Tr( \pi_1(g) ) Tr ( \pi_2(g^{-1}) )$ should really be thought of as $\sum_{g \in G} Tr(\pi_1(g) \otimes \pi_2^*(g))$, where $\pi_2^*(g)$ is the dual representation. In terms of matrices the dual representation looks like $\pi_2^*(g) = \pi_2(g^{-1})^\top$, and the tensor product is given by the Knonecker product of matrices.

Recall that if $\pi$ is any representation then $\frac{1}{|G|} \sum_{g \in G} \pi(g)$ is a $G$-equivariant projection onto the subspace of $G$ invariant vectors. (To see that check that it squares to itself so it's a projection, that it commutes with every $g \in G$, that it fixes all $G$ invariant vectors, and that any vector in the image is $G$ invariant.) In particular the trace of this operator gives the multiplicity of the trivial representation in $\pi$.

Now to see why I wrote it that funny way: We can recover the usual orthogonality for characters by applying the above to $\pi = \pi_1 \otimes \pi_2^*$ and noting that $\pi_1 \otimes \pi_2^*$ has a copy of the trivial representation iff $\pi_2 = \pi_1$, in which case it has multiplicity 1. This is because tensor products and duals are defined so that $Hom_G(\pi_1 \otimes \pi_2^*, \mathbb{C}) = Hom(\pi_1, \pi_2)$ and we know there is a map between two irreducibles iff they are isomorphic, in which case there is a one dimensional space of maps (that is Schur's lemma).

Let's just not take trace. $\frac{1}{|G|} \sum_{g \in G} \pi_1(g) \otimes \pi_2^*(g)$ is the projection onto the invariant subspace of $\pi_1 \otimes \pi_2^*$, and therefore is $0$ if $\pi_2 \ne \pi_1$. This is true for any irreps $\pi_1$ and $\pi_2$, regardless of their sizes.

To get the version of statement you want, with the product of matrices rather than the tensor product, just note that the multiplication map $Mat_{n \times n} \times Mat_{n \times n} \to Mat_{n \times n}$ is bilinear so it corresponds to a linear map $Mat_{n \times n} \otimes Mat_{n \times n} \to Mat_{n \times n}$ and the sum we care about is the image of $\frac{1}{|G|} \sum_{g \in G} \pi_1(g) \otimes \pi_2^*(g) = 0$ under a linear map.