Assume H,K are Hilbert spaces and $\pi : U \to B(K)$ is a a group representation (respectively a group-like unitary representation) where U is a group of unitary operators in B(H) (respectively a group-like unitary system in B(H)).
Let $S \subset K$ be an invariant subspace of $\pi$ (i.e. ($\forall u\in U)( \forall x \in S) : \pi(u)(x) \in S$)
Let $P : K \to K$ be the orthogonal projection on $S$.
What does the subrepresentation of $\pi$ defined by P, denoted by $\pi | P$, mean ?
I figured it out. It means the subrepresentation $\pi|S : U \to B(S)$.