$\ T: \mathbf R^5 \rightarrow \mathbf R^5 , \ T^2 =0$ and let $U$ be a subspace of $\ T $ and $\dim U = 3 $
prove: $\ U \cap \ker T \not= \{0\} $
If I understand correct then $w \in \operatorname{im}T, \ v \in \mathbf R^5, \ T(v) = w,\ T(w) = T(T(v)) = 0 $
so $\operatorname{im}T \subseteq \ker T$ and therefore $\dim(\operatorname{im}T) \leq \dim(\ker T) $ (not sure if it is true?)
and because $\dim(\mathbf R^5) = 5 $ so $\dim(\ker T) \geq 3 $
$\dim(\ker T) + \dim(U) = 6 > \dim(\mathbf R^5) $ it must be that $\ U \cap\ker T \not = \{0 \}$
I feel like i'm way off here..
Your reasoning looks good, but it should be $\dim(\ker T) + \dim(U) \ge 6 $, as you cannot guarantee equality.
If you'd like to justify your conclusion more carefully, consider proving that $ \dim(U) + \dim(V) = \dim(U + V) + \dim(U \cap V) $, for U and V subspaces of $ \mathbf R^n $ (or more generally, subspaces of some vector space W. EDIT: I should have said for some finite dimensional vector space W). If you have this fact, can you figure out how to conclude the result you want?