let p be prime. prove there is a group representation of $\Bbb Z/p\Bbb Z$ that is not decomposable over a field $\Bbb F_p$
for similar and simpler questions i showed homomorphism $\mu$ such that
$$\mu(x)(v) = \begin{pmatrix} 1 & x \\ 0 & 1 \\ \end{pmatrix} \begin{pmatrix} a \\ b \\ \end{pmatrix}$$
and then showed invariant field
$L = \begin{pmatrix} a \\ 0 \\ \end{pmatrix}$
that does not have an invariant complement.
I'm not sure if thats the correct way and if so then what is the explanation for that.
Any help will be appreciated.
You have the right idea. Let $\mu : \Bbb Z/p\Bbb Z \to GL_2(\Bbb F_p)$ such that $$\mu(x)=\begin{pmatrix} 1 & x \\ 0 & 1 \\ \end{pmatrix}$$
A non-trivial invariant subspace $L$ has dimension $1$, generated by some $(a,b)$.
You can show that $L$ must be generated by $(1,0)$ [see below]. So the only non-trivial invariant subspace is $L = \{(a,0) \mid a \in \Bbb F_p\}$
But then your vector space $V=\Bbb F_p^2$ is not the direct sum of $L$ with itself. More precisely, if $V$ was a direct sum of non-trivial invariant subspaces $$V= \bigoplus\limits_{i=1}^r W_i$$ then $r>1$ and $W_i=L$ for all $i$. But this can't happen because we have for instance $W_1 \cap W_2 = L \cap L = L \neq \{0\}$, so the sum can't be a direct sum.
Here is how you can show that $L = \langle (a,b) \rangle$ must be generated by $(1,0)$.
It is necessary for $L$ to be invariant under $\mu$ that $$\mu(1)(a,b)= \begin{pmatrix} 1 & 1 \\ 0 & 1 \\ \end{pmatrix} \begin{pmatrix} a \\ b \\ \end{pmatrix} = \begin{pmatrix} a+b \\ b \\ \end{pmatrix} = (a+b,b)= k(a,b)=(ka, kb) \in L$$ for some $k \in \Bbb F_p$. Then you have $a+b=ka$ and $b=kb$.
Conclusion: $L$ is generated by $(a,0)$ with $a≠0$, so it is also generated by $(1,0)$.