So I was reading Georgi's Lie Algebras in Particle Physics. On page 5, he said:
A representation is reducible if it has an invariant subspace, which means that the action of any $D(g)$ on any vector in the subspace is still in the subspace. In terms of the projection operator $P$ onto the subspace, this condition can be written as $$P D ( g ) P = D ( g ) P \forall g \in G$$
I do understand the intuition of reducible representation (I think it means the representation is, well, still a representation inside the subspace alone). But what I don't understand is the condition written above.
I think it should've said something like: after acting $D(g)$ on the projected vector $P|a>$, $P'|a>=D ( g ) P |a>$ is still in the subspace $S$, right? So I can formulate it as $D ( g ) P |a> \in S, \forall g \in G$, with $|a>$ be an arbitrary vector that these operators on, and $S$ be the vector space of those that are projected vectors $P|a>$, in other words $S \ni P|a>$? Why is the condition written like that? Thank you!
Your understanding of reducible representations is correct. The issue is with the role of the projection operator.
It is true that they are saying "For every $|a\rangle$ and every $g$ we have that $D(g)P|a\rangle \in S$ where $S$ is the subspace onto which $P$ is the projection.
What they then use is that the statement $|x\rangle \in S$ is equivalent to the statement $|x\rangle = P|x\rangle$. Think about it: for every $|x\rangle \in S$ the operator $P$ just leaves it as it was, and for every $|x\rangle \not\in S$ the operator $P$ maps it to a different vector. So writing $P|x\rangle = |x\rangle$ is the same as writing $|x\rangle \in S$ and that is what they do for the special case of $|x\rangle = D(g)P|a\rangle$.