I'm reading Linear Algebra by Peterson, and the excercise on p.31 reads as follows:
Let $L: W \to V$ be a linear operator and $V$ a vector space over $\mathbb{F}$. Show that if $M \subset V$ is $L$-invariant and $p \in \mathbb{F}[t]$ then $M$ is also invariant under $p(L)$.
Well, it doesn't seem to be the case, but the book is asking me to show it, so I must be going wrong somewhere.
So, we know that for $m_1, m_2 \in M$ $$ L(m_1) = m' \implies m' \in M $$
but we don't know if $$(\alpha m_1 + \beta m_2) \in M.$$
I could imagine that maybe $L$ doesn't take any $m$ outside $M$ but scalar multiplication does.
For example, Suppose we have $\mathbb{F} = \mathbb{R}, V = \mathbb{R}^2,$ and $M = \{(\alpha,\alpha+1) : \alpha \in \mathbb{R}\}$. Define $L(v) = v + (1,1)$.
Then for $$w \in M, \quad w = (\beta, \beta + 1),$$ clearly $$L(w) = (\beta + 1, \beta + 2) \in M$$ and yet $$2(\beta, \beta + 1) = (2\beta, 2\beta + 2) \notin M$$ So it would seem that $M$ is invariant under $L$. And yet if $p \in \mathbb{R}[t]$ is some polynomial, say $p = \gamma_0 t$, then if $\gamma \neq 1$, $M$ is not invariant under $p(L)$ since $$ p(L)(w) = \gamma_0(L(w)) =\big( \gamma (\beta + 1), \gamma (\beta + 1) + \gamma 1 \big) $$ Now to be precise I think that $L$ is a linear operator for $V$, but not for $M$, since $M$ won't admit $L(\gamma w)$. If we say $X$ is invariant under $L$ does it require $L$ to be a linear operator for $X$? I don't think so...
As Hagen mentioned, your $L$ is not linear. But in general, $M\subseteq V$ being invariant for a linear map $L \colon V \to V$ does not imply $M$ is a subspace, as for example every $M\subseteq V$ is invariant under $L = \mathrm{id}_V$. But one is often interested in invariant subspaces and this is not a restriction, as the following holds:
Lemma. Let $M \subseteq V$ be a subset, $L\colon V \to V$ linear. If $M$ is $L$-invariant, the generated subspace $\mathrm{span}\, M$ is also.
Let $v \in \mathrm{span}\, M$. Then there are $m_i \in M$, $\lambda_i \in \mathbb F$ such that $v = \sum_i \lambda_i m_i$. As $M$ is $L$-invariant, we have $Lm_i \in M$. Since $L$ is linear $$ Lv = \sum_i \lambda_i\,Lm_i \in \mathrm{span}\, M $$ So $\mathrm{span}\, M$ is invariant.