From Simon's "Operator Theory"
A representation of a topological group $\rho:G\to\mathcal{L}(H)$ on an Hilbert space it is said to be irreducible if the only subspaces $H'\subset H$ such that $\rho(g)H'\subset H'$ are $H'= \{0\} $ and $H'=H$.
What seems strange to me is to not require $H'$ to be a closed subspace of $H$.
Following Simon's definition, if $\rho$ is reducible then $H= H'\oplus W$ (as a vector space) but $\rho|_{H'}:H'\to H'$ can also not be a representation since maybe $H'$ is not an Hilbert space but only a pre-Hilbert space.
Do I misinterpret the definition or there is a way to prove that $H'$ would be closed in either case?
On the other hand if I assume $H'$closed in the definition, proving Schur's Lemma became more difficult. Indeed if $\varphi:V\to W$ is a $G$-module homomorphism of irreducible representations, then $Ker \varphi$ is an invariant subspace (so the map is null or injective), $Im \varphi$ is invariant but maybe is not closed, so I can only say that $\overline{Im\varphi}= W$ or $\varphi=0$. And I can't conclude that $\varphi$ is $0$ or an isomorphism (since it can be not closed) .