Let $\Phi:\mathbb{R^3}\longrightarrow \mathbb{R}$ be a quadratic form represented by the matrix $B$=$\begin{pmatrix} 0 & -3 & -3\\ -3 & 0 & -3\\ -3 & -3 & 0 \end{pmatrix}$. I have to find a matrix $C\in \mathrm{SO}(3)$ such that $B=C\Delta C^{-1}$, where $\Delta$ is a diagonal matrix.
What I've tried: We know that if $C\in \mathrm{SO}(3)$, then $C^{-1}=C^t$. The eigenspaces of our quadratic form (which is diagonalizable because every symmetric matrix is diagonalizable) are $V_{-6}=\mathrm{Span}((1,1,1))$ and $V_{3}=\mathrm{Span}((-1,1,0),(-1,0,1))$.
Now we know that the matrix $C^{-1}$ is a matrix whose columns are the eigenvectors of $V_{-6}$ and $V_{3}$. So, the matrix $C$ is the inverse of $C^{-1}$, with $\mathrm{det}(C)=1/3$ and $C C^{-1}=CC^t=I_{3}$.
The last point of this exercise is to find a basis that reduces $\Phi$ to its canonical form. In any case I solved this, I made lot of confusion but now it's all clear.
I'm not sure what you did but perhaps you did Gramn Schmidt to all the vectors and this screws orthogonality must of the times.
You take $\;u_1=\frac1{\sqrt3}(1,1,1)=\;$ an orthonormal basis of $\;V_{-6}\;$ , and then you apply GS on the basis of $\;V_{-3}\;$, obtaining
$$u_2=\frac1{\sqrt2}(-1,0,1)\;,\;\;\frac1{\sqrt6}(-1,2,-1)$$
and now you can chech $\;\{u_1,u_2,u_3\}\;$ is an orthonormal basis and with it you can construct your $\;C\;$ , and orthogonal matrix.
The above is specially easy because of what I commented before: $\;V_{-3}\perp V_{-6}\;$ ...