Assuming a linear transformation $T : \mathrm{V \rightarrow V}$, let $\mathrm{W \subset V}$ so that $\mathrm{W} \ne \{0\}$.
If $T(\mathrm{W}) \subseteq \mathrm{W}$ then $\mathrm{W}$ is $T$ invariant, and can be denoted $T|_W : \mathrm{W \rightarrow W}$ such that for $\forall w \in \mathrm{W},\quad T|_W(w) = T(w)$
I believe that I understand most of the definitions and implications associated with this term, however I seem to have a problem with the final equation. I'll elaborate:
let $\mathrm{V = \mathbb{R}_3[x]}$ and $\mathrm{B} = \{x^2, x+1, 1\}$ be a base of mentioned vector space such that:
$\mathrm{T}(x^2) = x^2\mathrm{,\quad} \quad\mathrm{T}(x+1) = x + 1\mathrm{,\quad} \quad\mathrm{T}(1) = 1$
$[T]_B = \begin{pmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} = I_3$
We can see that $\mathrm{W} = span\{x+1,1\}$ maintains the conditions and definitions above. The following also occurs:
$[T|_W]_B = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = I_2$
I understand the equation $T|_W(w) = T(w)$ when the representation is over $\mathrm{\mathbb{R}_3[x]}$, specifically $\mathrm{\mathbb{R}_2[x]}$ in our case. However, how is it interpreted when we deal with matrix representation or coordinate vector representation, considering we identify them with one another when dealing with invariant sub-spaces, and their dimensions aren't equal.
Your question (i.e. your final sentence) is not very clear, but if I understand it correctly, it can be answered with block matrices:
Let $V$ be a linear space with base $B=(e_1,\dots,e_n)$ and for some $p\le n,$ let $W$ be the subspace spanned by $B'=(e_1,\dots,e_p).$
$W$ is $T$-invariant iff the matrix of $T$ in $B$ is of the form $$\begin{pmatrix}P&Q\\0&R\end{pmatrix}$$ where $P$ is a $p\times p$ matrix, which is then the matrix of $T_{|W}$ in $B'.$