I have some doubts. Thanks in advance for taking the time to read.
$ \textbf {DEF:} $ A representation $ (\Pi, \mathscr{H}) $ of $ G $ in a space Hilbert $ \mathscr{H} $ is said $ \textbf {irreducible} $ if is decomposed $\mathscr {H} =\mathscr {H} _1 \dot {+} \mathscr{H}_2$ as a direct sum of two invariant subspaces implies that $\mathscr{H}_1 = {\{0}\}$ or $\mathscr{H}_1 = \mathscr{H}$.
I want to demonstrate Schur's Lemma.
$ \textbf {(Schur's Lemma)} $ Let $ \Pi $ be a representation ($ \mathbb {C} $ -linear) of a group of matrix Lie $ G $ on a vector space of finite dimension $ V $. Then:
$ i) $ If $ \Pi $ is irreducible and $ T: V \to V $ is an interlace operator ($ \mathbb {C} $ -linear), that is, $$ T \Pi (g) = \Pi (g) T $$ for all $ g \in G $. Then exists $ \lambda \in \mathbb {C} $ such that $ T = \lambda Id $ ($ Id $ denotes the identity mapping in $ V $).
$ ii) $ If $ T: V \to V $ is an interlace operator and for all $ g \in G $, $ T \Pi (g) = \Pi (g) T $ implies $ T = \lambda Id $ , then $ \Pi $ is irreducible.
I have tested this motto with the following definition of irreducible representation:
$ \textbf {DEF:} $ A representation $ (\Pi, \mathscr {H}) $ of $ G $ in a Hilbert space $ \mathscr {H} $ is said $ \textbf {irreducible} $ if the only subspaces closed G-invariants are $ {\{0}\} $ and $ \mathscr {H} $.
But my teacher says that this definition is only valid for unitary representations. So for generality it is better to use the definition given above. How can I demonstrate Schur's Lemma in this regard?
For most purposes, the second definition of "irreducible" is what you want: there are no proper topologically-closed $G$-stable subspaces.
Yes, this is a stronger requirement than the first, as can already be seen from a single operator $T=\pmatrix{\lambda & 1 \cr 0 & \lambda}$ on a two-dimensional vector space: there is a proper $T$-stable subspace, but that subspace does not have a $T$-stable complement. But/and it is almost always relevant/important to consider this two-dimensional example as most-definitely not irreducible.
There is an auxiliary notion of "complete reducibility", which among other things includes the idea that if there is a stable closed subspace, then it should have a stable closed complement. As the two-dimensional example already shows, a representation space can fail to be irreducible, yet not be completely reducible. Yes, we'd often want to avoid this, but it is not at all universally possible, e.g., in certain non-unitary representations.
Also, this is not a pathology: looking at representations of $G=SL_2(\mathbb R)$, the (unitary!) holomorphic discrete series are subrepresentations of (non-unitary, non-unitarizable) principle series representations, and do not have complements. (These principal series are also not repns on Hilbert spaces... even though they do contain Hilbert space subrepresentations...)