The question is, does the fact $$ \left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 0& 0\\ 0 & 1 &0 \end{array}\right)^{2}=0, \left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 0& 0\\ 1 & 0 &0 \end{array}\right)^{2}=0$$ influence $SU(3)$'s irreducible representation?
Background: To simplify I am working over the case of $SU(3)$. For all $sl_{n}\mathbb{C}$ any irreducible representation is generated by repeated applying negative weight elements to the highest weight vector $v$ (standard reference, Serre). But when I try to draw the weight diagram starting from any given highest weight vector $v$ for $H$, say (10, 20), then repeating applying the negative weight element like this one $ \left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 0& 0\\ 0 & 1 &0 \end{array}\right)$ always give me zero because $ \left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 0& 0\\ 0 & 1 &0 \end{array}\right)^{2}=0$. Now elementary analysis showed that $v$'s image under the action of $ \left(\begin{array}{ccc} 0 & 0 & 0\\ 0 & 0& 0\\ 0 & 1 &0 \end{array}\right)$ and $ \left(\begin{array}{ccc} 0 & 0 & 0\\ 1 & 0& 0\\ 0 & 0 &0 \end{array}\right)$ actually generates $V$. So $V$'s weight diagram must be very small, which is counter-intuitive for me. I am hoping I may claim there exists irreducible subrepresentation of $sl_{2}\mathbb{C}$ by similar arguments using either of the matrices repeatedly until the image vanishes. I think I must be confused with something very basic, like the embedding of $sl_{3}\mathbb{C}$ in $gl_{n}\mathbb{C}$ or something.
The short answer to your confusion is that what you think should hold, $\rho(xy)=\rho(x)\rho(y)$, does indeed hold for the representation of the Lie group, but not for the representation of the Lie algebra (indeed the product $xy$ makes no sense in a general Lie algebra). What holds for the Lie algebra is obtained by differentiation from what holds for the Lie group, and this gives the commutator relation of Jyrki's comment. Also note that for $G=SU(3)$, the Lie algebra $sl_{3}\mathbb{C}$ is the complexified Lie algebra; the nilpotent matrices you wrote do not live in the real Lie algebra of the compact real group $SU(3)$. The fact that everything (I mean Lie group and algebra, not the universal enveloping algebra) embeds in the same matrix ring is really a red herring I think.