So the question is to prove whether this matrix M is orthogonal?
$$ M = \begin{bmatrix} a & k & k \\ k & a & k \\ k & k & a \\ \end{bmatrix} $$
My attempt to find the inverse of M :
$$det(M)=(a+2k)(a-k)(a-k)$$
The inverse of $M$ is :
$$ \begin{matrix} (a+k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k))\\ (-k)/((a-k)(a+2k)) & (a+k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k))\\ (-k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k)) & (a+k)/((a-k)(a+2k))\\ \end{matrix} $$
$M$ is orthogonal if $M^T = M^{-1}$
So I end up with a system of 2 equations to solve :
$a=\dfrac{a+k}{(a-k)(a+2k)}$ and $k=\dfrac{-k}{(a-k)(a+2k)}$
and now I am stucked.
I would really appreciate feedbacks about whether my steps are all correct and if yes, what to do next ?
Since $M$ is symmetric, $M=M^T$, so to check orthogonality, we compute $MM^T=M^2$ and get $$ \begin{bmatrix} a^2+2k^2&2ak+k^2&2ak+k^2\\ 2ak+k^2&a^2+2k^2&2ak+k^2\\ 2ak+k^2&2ak+k^2&a^2+2k^2 \end{bmatrix} $$
Therefore, you need $2ak+k^2=0$ and $a^2+2k^2=1$. Factoring the first equation, we have $k(2a+k)=0$. Therefore, either $k=0$ or $k=-2a$.
When $k=0$, the second equation simplifies to $a^2=1$, so $a=\pm 1$ gives orthogonality.
When $k=-2a$, the second equation simplifies to $9a^2=1$, so $a=\pm\frac{1}{3}$ with $k=\mp \frac{2}{3}$ gives orthogonality.