I'm not quite sure how to start this exercise. What are the steps I should follow in order to solve it?
For which values of $k$ the matrix is not diagonalizable over $\Bbb R$?
$$\begin{pmatrix} k+3 & 0 & 0 \\ -k-3 & k & k+3 \\ -k-3 & k & k+3 \end{pmatrix}$$
This is my first answer so I apologize if it's not the best looking answer.
Let $A = \begin{pmatrix} k+3 & 0 & 0 \\ -k-3 & k & k+3 \\ -k-3 & k & k+3 \end{pmatrix}$
We'll find the eigenvalues of the matrix:
$ 0=|A - \lambda I| = \begin{vmatrix} k+3 & 0 & 0 \\ -k-3 & k & k+3 \\ -k-3 & k & k+3 \end{vmatrix}$
$= (k+3-\lambda)(k^2+3k-\lambda k - \lambda k - 3\lambda + \lambda ^2-k^2-3k)= \lambda(k+3-\lambda)(\lambda - (2k+3)) $
Therefore:
$ \lambda_1 = 0, \lambda_2 = k+3, \lambda_3 = 2k+3 $
Now, we'll find the corresponding eigenvectors:
For $\lambda_1 = 0$:
$ (A-0*I) = \begin{pmatrix} k+3 & 0 & 0 \\ -k-3 & k & k+3 \\ -k-3 & k & k+3 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} k+3 & 0 & 0 \\ 0 & k & k+3 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} k+3 & 0 & 0 \\ 0 & k & k+3 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} (k+3)x=0 \\ ky=(-k-3)z \end{cases} \Rightarrow \begin{cases} x=0 &(k \ne -3) \\ y=\frac{-k-3}{k}z & (k \ne0) \\ z= Free \end{cases} \Rightarrow \begin{pmatrix} 0\\\frac{-k-3}{k}\\1\end{pmatrix} $
For $\lambda_2 = k+3$:
$ (A-(k+3)*I) = \begin{pmatrix} 0 & 0 & 0 \\ -k-3 & -3 & k+3 \\ -k-3 & k & 0 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 0 & 0 & 0 \\ 0 & -3-k & k+3 \\ -k-3 & k & 0 \end{pmatrix}$
$\begin{pmatrix} 0 & 0 & 0 \\ 0 & -3-k & k+3 \\ -k-3 & k & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} (-k-3)y=(-k-3)z \\ (-k-3)x=-ky \end{cases} \Rightarrow \begin{cases} x=\frac{-k}{-k-3}z &(k \ne -3) \\ y=z \\ z= Free \end{cases} \Rightarrow \begin{pmatrix} \frac{k}{k+3}\\1\\1\end{pmatrix} $
For $\lambda_3 = 2k+3$:
$ (A-(2k+3)*I) = \begin{pmatrix} -k & 0 & 0 \\ -k-3 & -k-3 & k+3 \\ -k-3 & k & -k \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} -k & 0 & 0 \\ -6 & -3 & 3 \\ -3 & k & -k \end{pmatrix}$
$\begin{pmatrix} -k & 0 & 0 \\ -6 & -3 & 3 \\ -3 & k & -k \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} -kx=0 \\ -6x-3y+3z=0 \\ -3x+ky-kz = 0 \end{cases} \Rightarrow \begin{cases} x=0 & (k \ne 0)\\ y=z \\ z= Free \end{cases} \Rightarrow \begin{pmatrix} 0\\1\\1\end{pmatrix} $
Note: Since if $\lambda_3 = 0$ there will be repeated eigenvalues ($\lambda_1=0$), we'll check what happens if $k=-\frac{3}{2}$.
Therefore, for $k=0,-3$ the matrix is suspected to be not diagonalizable.
We'll check these values:
$\color{blue}{k=0}:$
$A_1 = \begin{pmatrix} 0 & 0 & 0 \\ -3 & 0 & 3 \\ -3 & 0 & 3 \end{pmatrix}, \lambda_1 = 3, \lambda_2=0$
For $\lambda_1 = 3$:
$(A-3I) = \begin{pmatrix} 0 & 0 & 0 \\ -3 & -3 & 3 \\ -3 & 0 & 0 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=0 \\ y=z \\ z=Free \end{cases} \Rightarrow \begin{pmatrix} 0\\1\\1\end{pmatrix} $
For $\lambda_2 = 0$:
$(A-0*I) = \begin{pmatrix} 3 & 0 & 0 \\ -3 & 0 & 3 \\ -3 & 0 & 3 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=0 \\ y=Free \\ z=0 \end{cases} \Rightarrow \begin{pmatrix} 0\\1\\0\end{pmatrix} $
The number of eigenvectors (2) is different from the number of matrix rows (3). Therefore, the matrix is not diagonalizable for $k=0$.
$\color{blue}{k=-3}:$
$A_2 = \begin{pmatrix} 0 & 0 & 0 \\ 0 & -3 & 0 \\ 0 & -3 & 0 \end{pmatrix}, \lambda_1 = 0, \lambda_2=-3$
For $\lambda_1 = 0$:
$(A-0*I) = \begin{pmatrix} 0 & 0 & 0 \\ 0 & -3 & 0 \\ 0 & -3 & 0 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}$
$\begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=Free \\ y=0 \\ z=Free \end{cases} \Rightarrow \begin{pmatrix} 1\\0\\0\end{pmatrix}, \begin{pmatrix} 0\\0\\1\end{pmatrix} $
For $\lambda_2 = -3$:
$(A-(-3)*I) = \begin{pmatrix} 3 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & -3 & 3 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 1 & 0 & 0 \\ 0 & -1 & 1 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 & 0 \\ 0 & -1 & 1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=0 \\ y=z \\ z=Free \end{cases} \Rightarrow \begin{pmatrix} 0\\1\\1\end{pmatrix} $
The number of eigenvectors (3) equals to the number of matrix rows (3). Therefore, the matrix is diagonalizable for $k=-3$.
$\color{blue}{k=-\frac{3}{2}:}$
$A_3 = \begin{pmatrix} \frac{3}{2} & 0 & 0 \\ -\frac{3}{2} & -\frac{3}{2} & \frac{3}{2} \\ -\frac{3}{2} & -\frac{3}{2} & \frac{3}{2} \end{pmatrix}, \lambda_1 = 0, \lambda_2=\frac{3}{2}$
For $\lambda_1 = 0$:
$(A-0*I) = \begin{pmatrix} \frac{3}{2} & 0 & 0 \\ -\frac{3}{2} & -\frac{3}{2} & \frac{3}{2} \\ -\frac{3}{2} & -\frac{3}{2} & \frac{3}{2} \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=0 \\ y=z \\ z=Free \end{cases} \Rightarrow \begin{pmatrix} 0\\1\\1\end{pmatrix} $
For $\lambda_2 = \frac{3}{2}$:
$(A-\frac{3}{2}*I) = \begin{pmatrix} 0 & 0 & 0 \\ -\frac{3}{2} & -3 & \frac{3}{2} \\ -\frac{3}{2} & -\frac{3}{2} & 0 \end{pmatrix} \rightarrow \cdots \rightarrow\begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}$
$\begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & -1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} x\\y\\z\end{pmatrix} = \begin{pmatrix} 0\\0\\0\end{pmatrix} \Rightarrow \begin{cases} x=-z \\ y=z \\ z=Free \end{cases} \Rightarrow \begin{pmatrix} -1\\1\\1\end{pmatrix} $
The number of eigenvectors (2) is different from the number of matrix rows (3). Therefore, the matrix is not diagonalizable for $k=-\frac{3}{2}$.
Finally, we can say that the matrix is not diagonalizable over $\Bbb R $ for $k=0, -\frac{3}{2}$ (only).