I am trying to find all matrices which solve the matrix equation
$$M^2 -3M +3I=0$$
Since this doesn't factor I tried expanding this in terms of the coordinates of the matrix. It also occurs to me to put it into "vertex" form:
$$M^2 - 3M + \frac{9}{4}I+\frac{3}{4}I=0$$
$$(M-\frac{3}{2}I)^2 = -\frac{3}{4}I$$
but this doesn't look much better.
What I found from expanding by coordinates was, if $M=\pmatrix{a & b \\ c & d}$ then
$$\pmatrix{a^2+bc -3a + 3& ab + bd - 3b \\ ac+cd-3c & bc+d^2-3d+3} = \pmatrix{0&0\\0&0}$$
From the off-diagonal entries I get that either
$$a+d-3=0$$
or
$$b=c=0$$
If $a+d-3\not=0$ then $a^2-3a+3=0$ and likewise for $d$. Then we get more cases for $a$ and $d$.
If $a+d-3=0$ the upper-left is unchanged and the lower-right is
$$bc + (3-a)^2-3(3-a)+3 = 0$$
which simplifies to the same thing from the upper-left and so is redundant. In the off-diagonals
$$ac+c(a-3)-3c = 0 \Rightarrow $$ $$2ac-6c = 0$$
We again get cases, and I suppose after chasing cases enough you get the solution set.
However, it just feels like this can't be the intended solution given how tedious and uninformative all of this case-chasing is. Is there some bigger idea I'm missing?
You already found the "completion of the square" $$ \left( {M - {3 \over 2}I} \right)^{\,2} = - {3 \over 4}I $$
Then you can write $$ \left( {i{2 \over {\sqrt 3 }}\left( {M - {3 \over 2}I} \right)} \right)^{\,2} = X^{\,2} = I $$
So we are essentially looking for the square roots of the unit matrix, also complex, or for the square roots of $- \, I$.
You can find various papers dealing with this subject, for example this related post or this thesis.
-- p.s. --
I thought you were interested in the general case of $n \times n$ matrices.
If your question is limited to $2 \times 2$ then the $\sqrt{\pm I}$ is easily found on the net (e.g.,see the hint on Pauli matrices).