let E be $Z/pZ$-vector space of dimension n. calculate number of linear maps from $E\to E$ that satisfy $f^2=0$ there is a hint that suggest that to consider the null space and subspace in direct sum with it.
I know that the number subspaces of a given dimension is given a previous post: How to count number of bases and subspaces of a given dimension in a vector space over a finite field? but I don't know where to start, any help please!
On
Based on the other answer's explanation, we may compute the total as follows: let $m = \lceil n/2\rceil$. Then in the notation of the post that you linked, the total will be $$ \sum_{j=m}^n \binom nj_p \binom{j}{n-j}_p \prod_{k=0}^{j-1}(p^j - p^k) $$
Let $K:=\operatorname{Ker}(f)$ and $I:=\operatorname{Im}(f)$.
$f^2=0$ is equivalent to $I\subseteq K$.
So you first pick a subspace $K$ of $E$ and then pick a subspace $I$ of $K$. The sets of functions you'll attain for different $K$ and $I$ are clearly disjoint so if you manage to count for each $I$ and $K$, you'll then just have to sum over all $I$ and $K$.
Now, suppose $I$ and $K$ fixed. Take some $H$ so that $E=H\oplus K$. Then it is equivalent to give the action of some $f$ on $E$ or to give it on both $H$ and $K$. Since the action on $K$ is trivial, giving the action of $f$ on $E$ is equivalent to giving its action on $H$, i.e. a function $H\to I$ that has image $I$. Since those functions are surjective and both spaces are or equal dimension, those functions will all be automorphisms. Now take $\varphi$ any (fixed) automorphism between $I$ and $H$. The application $( g : H \to I)\mapsto (\varphi \circ g : H \to H)$ is a bijection. So to count the morphisms when $I$ and $K$ are fixed, you just have to count automorphisms of $I$.