So this is the problem:
Let $A= \begin{bmatrix} 0 & 1 & 0 \\ 0 & 0 &-1 \\ 0 & 0 & 0 \\ \end{bmatrix}$ and $B= \begin{bmatrix} 0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ \end{bmatrix} $
(a) Show that $A^3=B^3=C^3=0$, where $C=\lambda A+\alpha B$, $\lambda,\alpha\in\mathbb R$.
(b) Does there exist an integer $k$ such that $(AB)^k=0$?
I know that $A^3$ and $B^3$ equal zero. That I already confirmed. But how does $C^3$ equal to zero? Also shedding light on b) would be appreciated.
On
The second question is simple: $AB$ is a diagonal matrix with diagonal entries $(1,-1,0)$, and clearly powers of this matrix are never zero.
For the first question it suffices to simply compute: $$ \lambda A+\mu B = \pmatrix{0&\lambda&0\\ \mu&0&-\lambda\\ 0&\mu&0},\quad (\lambda A+\mu B)^2= \pmatrix{\lambda\mu&0&-\lambda^2\\ 0&0&0\\ \mu^2&0&-\lambda\mu\\}, $$ which gives $$ (\lambda A+\mu B)^3=\pmatrix{0&0&0\\ 0&0&0\\ 0&0&0}. $$ I don't think there is a particularly insightful short-cut to this computation.
On
You could express $$(\lambda A+\alpha B)^3 = \lambda^3 A^3 + \alpha^3 B^3 + \lambda^2\alpha(A^2B+ABA+AB^2)+\lambda\alpha^2(AB^2+BAB+B^2A).$$ Knowing that $A^3=B^3=0$ simplifies this a tiny bit. But you still have to compute the matrices in parenthesis and check that they are equal to zero matrix.
$$C = \begin{bmatrix} 0&\lambda&0 \\\alpha&0&-\lambda \\0&\alpha&0 \end{bmatrix}$$
It's easy enough to prove that $C^3 = 0$ by solving it, as long as you don't mind dirtying your hands a little.
As for b), there's none, since $AB$ is a diagonal (nonzero) matrix.
Other than that, you can verify by doing the math that $(AB)^2 = D\ne 0$ and $(AB)^3 = AB$, therefore $(AB)^k$ is always $D$ if $k$ is even, and $AB$ if $k$ is odd.