X=(x,y,z)
Q(X) = $x^2 + 4xy + 6xz + 3y^2 +8yz +5z^2 $
I got by using completing the square method:
Q(X) = $(x+2y+3z)^2 - (y+2z)^2$
so as I learned now I do:
$u = x+2y+3z$
$v = y+2z$
$w = 0 $
$$\begin{pmatrix}u&\\v\\w\end{pmatrix}= \begin{pmatrix}1&2&3\\0&1&2\\0&0&0\end{pmatrix} \begin{pmatrix}x&\\y\\z\end{pmatrix} $$
but of course its impossible because the matrix is not invertible (generally the inverse matrix is the basis that gets the canonical form). where's my mistake? please try to explain me using what i did so far
Since $Q$ in terms of your new variables is simply $u^2-v^2$, its matrix form should be $$\begin{pmatrix}1&0&0\\0&-1&0\\0&0&0\end{pmatrix}\ .$$ It is not invertible, in fact also the matrix from in your original coordinate had nontrivial kernel: $$ \begin{pmatrix}1&2&3\\2&3&4\\3&4&5\end{pmatrix}\ ,$$ you can notice that the third row is obtained by subtrating the first row from the double of the sencond.
This simply means that your original quadratic forms is nondegenerate only on a lower dimensional subspace. And in particular this means that there is more than one "good" change of coordinates: you want a change of coordinates of the 3-space but your are interested only at the behaviour on a subspace. So you are free to choose how the change of coordinates behave in the complement of that subspace, with the only condition that it is an admissible change of coordinates (i.e. the matrix is invertible). Your choice of putting all zeros in the last row does not respect this condition.