Assume that $$S_1 = \begin{pmatrix}1&0&1\\ 0&1&0\\ 1&0&0\end{pmatrix}, S_2 = \begin{pmatrix}1&0&1\\ 0&-3&0\\ 1&0&0\end{pmatrix}, S_3 = \begin{pmatrix}0&0&0\\ 0&0&0\\ 0&0&1\end{pmatrix}.$$ Show that a finite set $S = \left \{ S_1, S_2, S_3 \right \}$ is an orthonormal system with respect to the inner product given by $$\left \langle A, B \right \rangle = tr \left ( A\cdot B \right ) = tr\left ( \begin{pmatrix}a_{1_1}&a_{1_2}&a_{1_3}\\ a_{1_2}&a_{2_2}&a_{2_3}\\ a_{1_3}&a_{2_3}&a_{3_3}\end{pmatrix} \cdot \begin{pmatrix}b_{1_1}&b_{1_2}&b_{1_3}\\ b_{1_2}&b_{2_2}&b_{2_3}\\ b_{1_3}&b_{2_3}&b_{3_3}\end{pmatrix}\right) = \\a_{1_1}b_{1_1}+a_{1_2}b_{1_2}+a_{1_3}b_{1_3}\\+a_{1_2}b_{1_2}+a_{2_2}b_{2_2}+a_{2_3}b_{2_3}\\+a_{1_3}b_{1_3}+a_{2_3}b_{2_3}+a_{3_3}b_{3_3}$$ for any $A$ and $B$ symmetric real-valued matrices of the size $3\times3$.
I know that the set is orthogonal - $\left \langle S_1, S_2 \right \rangle = 0$, $\left \langle S_1, S_3 \right \rangle = 0$, and $\left \langle S_2, S_3 \right \rangle = 0$. How do I now prove orthonormality?
You don't, since it isn't. For instance$$\langle S_1,S_1\rangle=\operatorname{tr}\begin{bmatrix}2 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 1\end{bmatrix}=4.$$