The polynomial is given: $f=x_1x_2-x_2x_3$. I have to construct a matrix for this form before and after a reduction to the canonical form.
I have a difficulty even at the first step, the matrix that I get is like this:
$$\begin{pmatrix}0 & 1/2 & 0 \\1/2 & 0 &-1/2 \\0 & -1/2 & 0\end{pmatrix}$$
So, following to that I compute the $\det(A-\lambda\cdot I)=0$ and get eigenvalues $\lambda_1=0$ and $\lambda_2=\frac{1}{\sqrt{2}}$. (The way I calculated the eigenvalues was just by computing the said determinant and equal the found expression to zero.)
So, what should I do next?
$$A = \begin{bmatrix}0 & 1/2 & 0 \\1/2 & 0 &-1/2 \\0 & -1/2 & 0\end{bmatrix}$$
$S_1 =0+0+0 = 0$, $S_2 = -\frac{1}{4}+0-\frac{1}{4} = -\frac{1}{2}$ , $S_3 = \det(A) = 0$
$$\lambda^3-S_1\lambda^2+S_2\lambda -S_3 = 0 \implies \lambda^3 - 0 - \frac{1}{2}\lambda-0=0$$
$$\lambda(\lambda^2 -\frac{1}{2}) = 0 \implies \lambda=0,\frac{1}{\sqrt2},\frac{-1}{\sqrt2}$$
So, the reference diagonal matrix is ,
Let $$Y = \begin{bmatrix}y_1 \\y_2 \\y_3\end{bmatrix}$$
So, the required canonical form is,
You've found $X_1$ and $X_3$ correctly.
But there's a mistake with your $X_2$ .
$X_2 = \begin{bmatrix}1\sqrt2 \\1 \\-1\sqrt2\end{bmatrix}$
Normalized vectors: $\frac{1}{\sqrt2}\begin{bmatrix}1\\0 \\1\end{bmatrix}$,$\frac{1}{\sqrt2}\begin{bmatrix}1\sqrt2 \\1 \\-1\sqrt2\end{bmatrix}$,$\frac{1}{\sqrt2}\begin{bmatrix}-1\sqrt2 \\1 \\1\sqrt2\end{bmatrix}$
$N = \frac{1}{\sqrt2}\begin{bmatrix} 1 & \frac{1}{\sqrt2}&-\frac{1}{\sqrt2}\\0 &1 &1\\1 &-\frac{1}{\sqrt2}&\frac{1}{\sqrt2}\end{bmatrix}$
$N^T = \frac{1}{\sqrt2}\begin{bmatrix} 1 & 0&1\\\frac{1}{\sqrt2} &1 &-\frac{1}{\sqrt2}\\-\frac{1}{\sqrt2} &1&\frac{1}{\sqrt2}\end{bmatrix}$
Now, $$AN = \frac{1}{\sqrt2}\begin{bmatrix}0 & 1/2 & 0 \\1/2 & 0 &-1/2 \\0 & -1/2 & 0\end{bmatrix}\begin{bmatrix} 1 & \frac{1}{\sqrt2}&-\frac{1}{\sqrt2}\\0 &1 &1\\1 &-\frac{1}{\sqrt2}&\frac{1}{\sqrt2}\end{bmatrix} =\frac{1}{\sqrt2}\begin{bmatrix}0&\frac{1}{2}&\frac{1}{2}\\0&\frac{1}{\sqrt2}&-\frac{1}{\sqrt2}\\0&-\frac{1}{2}&-\frac{1}{2}\end{bmatrix}$$
$$D = N^TAN = \frac{1}{2}\begin{bmatrix} 1 & 0&1\\\frac{1}{\sqrt2} &1 &-\frac{1}{\sqrt2}\\-\frac{1}{\sqrt2} &1&\frac{1}{\sqrt2}\end{bmatrix}\begin{bmatrix}0&\frac{1}{2}&\frac{1}{2}\\0&\frac{1}{\sqrt2}&-\frac{1}{\sqrt2}\\0&-\frac{1}{2}&-\frac{1}{2}\end{bmatrix} = \frac{1}{2}\begin{bmatrix}0 & 0 &0\\0&\sqrt2&0\\0&0&-\sqrt2\end{bmatrix}$$
Thus,