Let $S$ be defined as $$S:= \left\{ M \in \mathbb{F}_3^{2 \times 2} : M \text{ is orthogonal} \right\}$$ where $\mathbb{F}_3$ is a field with $\mathbb{F}_3 = \{ 0, 1, 2\}$. How many elements does the set $S$ have?
I would like some advice on how to approach/solve this problem.
Let $(x,y)^T$ be a column of $M$. Then $x^2+y^2=1$ and at least one of $x$ or $y$ is nonzero. Suppose $x$ is nonzero. Then $x^2=1,y^2=0$ and $(x,y)=\pm(1,0)$.
It follows that every column of $M$ is, up to a sign, a member of the standard basis. Therefore $M=PD$ for some permutation matrix $P$ and some invertible diagonal matrix $D$. Hence there are $(2!)(2^2)=8$ choices in total, namely, $$ \pmatrix{p&0\\ 0&q}\text{ and }\pmatrix{0&p\\ q&0} $$ for any $p,q\in\{-1,1\}$.
(Below is the old answer -- I misread the question and thought that $M$ was $3\times3$.)
Let $(x,y,z)^T$ be a column of $M$. Then $x^2+y^2+z^2=1$ and at least one of $x,y$ or $z$ is nonzero. Suppose $x$ is nonzero. Since $$ \text{every nonzero member of $\mathbb F_3$ squares to $1$},\tag{$\ast$} $$ we have $x^2=1$ and $y^2+z^2=0$. Hence $y$ and $z$ are either both zero or both nonzero. The latter is impossible, or else $(y/z)^2=-1$, which is a contradiction to $(\ast)$. Therefore $y=z=0$ and $(x,y,z)=\pm(1,0,0)$.
It follows that every column of $M$ is, up to a sign, a member of the standard basis. Therefore $M=PD$ for some permutation matrix $P$ and some invertible diagonal matrix $D$ (this decomposition is unique, for, if $P_1D_1=P_2D_2$, then $P_2^TP_1=D_2D_1^{-1}$ and hence both sides are equal to $I$). Hence there are $(3!)(2^3)=48$ choices in total.