I need to find a non-null matrix A in which A² = 0. It must be an order 3 matrix.
What I tried to do was split it in a table:
[A, B, C] * [A, B, C]
[D, E, F] * [D, E, F]
[G, H, I] * [G, H, I]
But didn't get very far, I got way too many equations to equal to 0, and got completely lost.
I must find a non-trivial answer, how can I do it?
Sometimes it is important to try and simplify your question, using your intuition and what you already know.
What does your intuition tell you? "This matrix could have a lot of zeros" comes to mind. Try the first matrix that comes to mind: One non-zero entry.
Now, think about this matrix: $$ \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix} $$
What happens when you multiply this matrix with itself? $$ \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix} \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix} $$
Every non-zero entry in this matrix, gets multiplied by a zero entry, so you see that: $$ \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix}\begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix}=\begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\0 & 0 & 0 \\\end{pmatrix} $$
Unfortunately, you will find problems when you try to add another non-zero entry (try it!), but at least you have produced an example of a (nilpotent) matrix.
Something more non-trivial than this example doesn't immediately come to mind, but it will involve lots of non-zero entries and cancellations, that is likely to happen.
If $A$ is a $3 \times 3$ matrix then $A$ is the matrix (in any basis), of a linear transformation from $\mathbb R^3$ to $\mathbb R^3$.
Therefore, $A$ has a kernel and a range. If $A^2 = 0$, then note that $\ker A$ contains the range of $A$, since $x =Ay \implies Ax = A^2 y = 0$.
Therefore, to construct any $A$ such that $A^2 = 0$, all we need to do is the following :
decide the kernel of $A$, $K$. This will be a subspace of $\mathbb R^3$. If we want $A$ to be non-trivial, we desire $K \neq \mathbb R^3$.
Find a basis for $K$, and extend this basis to the whole space. Call this basis $B$.
Define $A$ , a linear transformation, as follows : it will have kernel exactly $K$ : so send every element of $B$ belonging to $K$, to zero. Next, ensure that every other basis element goes to some non-zero element of the kernel. Then $K$ is exactly the kernel of $A$, and $A^2 = 0$.
For example, take the standard basis $e_1,e_2,e_3$, and let us say we want the kernel equal to the span of $e_1,e_2$. Define the linear tranformation $A$, which takes $e_1,e_2$ to zero and $e_3 \to e_1$. Then, $A$ is non-zero, but $A^2 = 0$. In standard basis, the matrix of $A$ is exactly the first matrix in this answer!