The usual way to denote RSS is $\sum\limits_{i=1}^n (Y_i-\hat{Y_i})^2$
By writing in the matrix form $Y=X \beta + \epsilon$. Then I know $\hat Y=PY$
Thus, the RSS=$|| (I_n-P)Y ||^2 = || (I_n-P)(Y-X\beta) ||^2= \epsilon^T (I_n-P)\epsilon$
How to see the above last two equal signs?