Could you help me with this question?
Let $\mathcal{g}$ be a nilpotent Lie algebra over $k$ and $\rho$ a representation of $\mathcal{g}$ in a finite-dimensional nonzero vector space $V$ over $k$. Suppose that det $\rho (X) = 0$ for all $X \in \mathcal{g}$. Show that there is a nonzero vector $v \in V$ such that $\rho(X)v = 0$ for all $X \in \mathcal{g}$.
Typically Engel's theorem on nilpotent Lie algebras is proved with the following Theorem:
Theorem: If $ρ \colon \mathfrak{g} \rightarrow \mathfrak{gl}(V )$ is a representation on a (non-zero) vector space $V$ such that $\rho(a)$ is a nilpotent linear transformation for any $a ∈ \mathfrak{g}$, then $V$ has a non-zero vector $v$ such that $\rho(\mathfrak{g})v = 0$.
For a proof, see for example Proposition $2.3$ here. This theorem is usually proved in a book on Lie algebras.
Edit: The question is exactly question $21$ in chapter $3$ of Varadarajan's book. The way I would prove it is by using the weight space decomposition for representations of nilpotent Lie algebras, i.e., Theorem $9.1$ here. Then the assumption $\det \rho(X)=0$ for all $X$ implies that $\lambda_i(x)=0$ in the theorem, so that all operators $\rho(X)$ are nilpotent. This follows first over the complex numbers, but of course then also holds over the real numbers. Now the claim concerning the common eigenvector follows by the above result. Note that the proof of Theorem $9.1$ uses Lie's theorem.