Let $$A = \frac{1}{15} \left( \begin{array}{rrr} 10 & 5 & 10 \\ 5 & -14 & 2 \\ 10 & 2 & -11 \end{array}\right)$$ Show that, $A \in \mbox{SO} (3)$ and find an orthonormal basis, so that $A$ have canonical form.
To show that $A$ is in $\mbox{SO} (3)$ , I have to show that $\det(A)=1$, because the matrices of $\mbox{SO} (3)$ are rotation matrices and have as determinant 1(I checked that and it is true) and $A$ is orthogonal.
To show tha $A$ is orthogonal I did it in two ways. The first way is to check $AA^{-1} = E_3$. This equality is fullfilled because $A$ = $A^{-1}$ and therefore $AA^{-1} = E_3$.
The second way I did, was to check if the column vectors are orthogonal i.e. $\ \ \ v_i \cdot v_j=0$ for $i \neq j .$ And this is also true.
Now I showed that $A \in \mbox{SO} (3)$.
I read a theorem in my book that states that columns and rows of orthogonal matrices form orthonormal bases. So we have now an orthonormal bases. I took the columns as an orthonormal bases. Nevertheless, I checked whether it is really an orthonormal basis. This is the case if the length of the column vectors of the matrix $A$ is equal to 1. And this is also fulfilled.
To construct the canonical form, I thought about what it might look like, and I came up with the following form
$$\mathbf{S}^\intercal A S = \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & \cos\alpha & -\sin\alpha \\ 0 & \sin\alpha & \cos\alpha \end{array}\right).$$
My goal is to find a matrix of this form. From here I have problems to reach the goal. Because I have not yet fully understood how I can determine the matrix $S$. Can someone give me a hint here?
Let $$ A = \frac{1}{15} \begin{bmatrix} 10 & 5 & 10 \\ 5 & -14 & 2 \\ 10 & 2 & -11 \\ \end{bmatrix}. $$ The characteristic polynomial of $A$ is $$ f(x) = \det {(xI - A)} = (x + 1)^2 (x - 1), $$ in which $I$ is the identity matrix. Hence the determinant of $A$ must be $1$. Note that $A = A^{\mathrm{T}}$. It is not hard to check that $$ A^2 = I = AA^{\mathrm{T}} = A^{\mathrm{T}}A. $$ Hence $A$ is indeed in $\mathrm{SO}(3)$. (Note that you have reached a correct conclusion for a wrong reason: "The first way is to check $AA^{-1} = E_3$.")
All eigenvalues are real, so the "canonical form" is a real diagonal matrix.
It is not hard to find eigenvectors of $A$.
Solving the system of linear equations $(A - I)X = 0$ yields $$ X = k \begin{bmatrix} 5 \\ 1 \\ 2 \\ \end{bmatrix}. $$ Put $$ X_1 = \frac{1}{\sqrt{30}} \begin{bmatrix} 5 \\ 1 \\ 2 \\ \end{bmatrix}. $$ It is easy to check that $X_1^{\mathrm{T}} X_1 = 1$ and that $AX_1 = 1X_1$.
Solving the system of linear equations $(A + I)X = 0$ yields $$ X = k \begin{bmatrix} -1 \\ 5 \\ 0 \\ \end{bmatrix} + \ell \begin{bmatrix} -2 \\ 0 \\ 5 \\ \end{bmatrix}. $$ We wish to find two solutions that are unit vectors and that are orthogonal. Put $$ Y_2 = 1 \begin{bmatrix} -1 \\ 5 \\ 0 \\ \end{bmatrix} + 0 \begin{bmatrix} -2 \\ 0 \\ 5 \\ \end{bmatrix} = \begin{bmatrix} -1 \\ 5 \\ 0 \\ \end{bmatrix} $$ and $$ Y_3 = k \begin{bmatrix} -1 \\ 5 \\ 0 \\ \end{bmatrix} + \ell \begin{bmatrix} -2 \\ 0 \\ 5 \\ \end{bmatrix} = \begin{bmatrix} -k-2l \\ 5k \\ 5l \\ \end{bmatrix}. $$ Solve the equation $Y_2^{\mathrm{T}} Y_3 = 0$ yields $$ \ell = -13k. $$ Put $k = 1$ and $\ell = -13$. Then $$ Y_3 = \begin{bmatrix} 25 \\ 5 \\ -65 \\ \end{bmatrix} = 5\begin{bmatrix} 5 \\ 1 \\ -13 \\ \end{bmatrix}. $$ It is easy to check that $AY_2 = -1Y_2$ and that $AY_3 = -1Y_3$ and that $Y_2^{\mathrm{T}} Y_3 = 0$. Put $$ X_2 = \frac{1}{\sqrt{26}} \begin{bmatrix} -1 \\ 5 \\ 0 \\ \end{bmatrix} $$ and $$ X_3 = \frac{1}{\sqrt{195}} \begin{bmatrix} 5 \\ 1 \\ -13 \\ \end{bmatrix}. $$ It is easy to check that $AX_2 = -1X_2$ and that $AX_3 = -1X_3$ and that $X_2^{\mathrm{T}} X_3 = 0$ and that $X_2^{\mathrm{T}} X_2 = X_3^{\mathrm{T}} X_3 = 1$.
Put $$ S = [X_1, X_2, X_3] = \begin{bmatrix} \displaystyle \frac{5}{\sqrt{30}} & \displaystyle \frac{-1}{\sqrt{26}} & \displaystyle \frac{5}{\sqrt{195}} \\ \displaystyle \frac{1}{\sqrt{30}} & \displaystyle \frac{5}{\sqrt{26}} & \displaystyle \frac{1}{\sqrt{195}} \\ \displaystyle \frac{2}{\sqrt{30}} & \displaystyle 0 & \displaystyle \frac{-13}{\sqrt{195}} \\ \end{bmatrix}. $$ It is easy to check that $S^{\mathrm{T}} S = S S^{\mathrm{T}} = I$ and that $AS = SD$, in which $$ D = \begin{bmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & -1 \\ \end{bmatrix}. $$ Hence $$ S^{\mathrm{T}} AS = S^{\mathrm{T}} SD = D. $$