I have a problem with this quadratic form: $$q(x,y,z) = x^2+ay^2+3z^2+2xy+2xz+2ayz$$
What I have to do is to clasify it depending on the value of $a$
I get stuck in this step:
$$A = \begin{pmatrix}1&1&1\\1&a&a\\1&a&3\end{pmatrix}$$ $$\begin{array}{rcl} \det(A-\lambda I_3) &=& (1-\lambda)(a-\lambda)(3-\lambda)+a+a-\left[a-\lambda+3-\lambda+a^2(1-\lambda) \right]\\ &=& (1-\lambda)(a-\lambda)(3-\lambda)+2a-\left[a+3-2\lambda+a^2(1-\lambda)\right]\\ &=& (1-\lambda)(a-\lambda)(3-\lambda)+2a-a-3+2\lambda-a^2(1-\lambda)\\ &=& (1-\lambda)(a-\lambda)(3-\lambda)+a-3+2\lambda-a^2(1-\lambda) \end{array}$$
I'm not sure if I understand your question, because, as far as I know, every quadratic form is "diagonalizable"; that is, can be transformed into a sum of squares with a linear change of variables.
I think you've mixed two different things: "diagonalization" of a (general) endomorphism, and "diagonalization" of a quadratic form. The first is not always possible (though it is in this case, see the end of my answer), the second always is.
Moreover, you have got an easy algorithm to do this: just perform elementary transformations on your matrix $A$, as if you were looking for a row echelon form, BUT, whenever you perform a row transformation you must do the same transformation with columns.
For instance, you can start substracting the first row from the second and the third, getting
$$ \begin{pmatrix} 1 & 1 & 1 \\ 0 & a-1 & a-1 \\ 0 & a-1 & 2 \end{pmatrix} $$
And now you have to do the same operations with columns:
$$ \begin{pmatrix} 1 & 0 & 0 \\ 0 & a-1 & a-1 \\ 0 & a-1 & 2 \end{pmatrix} $$
If you now substract the second row from the third one and do the same with columns, you'll get a diagonal form for your quadratic one:
$$ \begin{pmatrix} 1 & 0 & 0 \\ 0 & a-1 & 0 \\ 0 & 0 & 3-a \end{pmatrix} $$
So, after a changing of variables, your form becomes:
$$ q(\overline{x}, \overline{y}, \overline{z}) = \overline{x}^2 + (a-1)\overline{y}^2 + (3-a)\overline{z}^2 \ . $$
And if you want to know which change of variables gives you this reduced quadratic form, there is an easy way to obtain it. Namely, if
$$ X = \begin{pmatrix} x \\ y \\ z \end{pmatrix} $$
and $\overline{X}$ is the analogous vector for the new variables, then they are related through a linear relation $X = S\overline{X}$ and you can find $S$ as follows: add to your matrix $A$ the identity matrix $I$ like this:
$$ (A\ |\ I) \ . $$
Now, perform to $I$ the same row operations you do to $A$ (just row operations!). In the end, you'll get a matrix
$$ (D\ |\ S^t) $$
where $D$ is the previous diagonal one, up there, and the one appearing on the right is the transpose of the change of variables matrix. And the relations among all these matrices that have appeared here so far is:
$$ D = S^tA S \ . $$
Another way to do this, though with more involved computations: your matrix $A$ is a symmetric one, right? Well, from the endomorphism point of view, every diagonal real matrix diagonalizes and with an orthonormal basis of eigenvectors. So, you have a diagonal matrix $D$ and an orthogonal bases change matrix $S$ (hence, $S^{-1} = S^t$) such that the above relation between all of them holds.