If possible, compute the Jordan normal form of
$\begin{pmatrix}0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & a & b \end{pmatrix}\in\mathbb{R}^{3\times 3}$ with $a,b\in\mathbb{R}$.
In the case that $a,b=0$ the matrix already has Jordan normal form. However, the case that one or both, $a,b\neq0$ seems more complicated. How do I continue?
Edit: The eigenvalues are $b/2\pm\sqrt{b^2/4+a}$. Using this in order to find the kernel of $(A-\lambda_i)^j$ using Gaussian elimination doesn't seem like the intended approach. That is what I meant.
We can avoid computing generalized eigenvectors. Separate this into $3$ cases.
Case 1: $a = b = 0$. As you said, the matrix is already in Jordan form
Case 2: $a=0$, $b \neq 0$. The matrix is upper triangular, so we quickly see that its only eigenvalues are $0$ and $b$. The block associated with $b$ has size $1$, and the because the matrix $M$ has rank $2$, the block associated with $0$ has size $2$.
Now, if $a \neq 0$, then $M$ is block upper triangular with the form $$ M = \left[\begin{array}{c|cc} 0&1&0\\ \hline 0&0&1\\0&a&b\end{array}\right] $$
Case 3: If $a = -(b/2)^2 \neq 0$, then the lower-right block has a repeated eigenvalue, and its Jordan form consists of a single block. Thus, the overall Jordan form has a $0$ followed by a size-$2$ block.
Case 4: In the remaining cases, the lower-right block has eigenvalues that are distinct and non-zero. Thus, $M$ has distinct eigenvalues, which means that its Jordan form is diagonal.
As an alternative, it would suffice to note that $M$ is the transpose of the companion matrix associated with the polynomial $p(t) = t^3 - bt^2 - at$. It follows that its Jordan form consists of a single block of maximal size for each of the roots of $p$.