Find a coordinate transformation diagonalizing the quadratic form.
Interesting in answering number 2. So, here is my approach:- Step 1:- Write the matrix representation of the equation, that is
A=
1 1 3
1 1 -3
3 -3 -3
Step 2: After doing this I then use a regular 3x3 matrix diagonalisation procedure. That includes finding the eigen values and vectors, and using this formula :- D=P^-1(A)P. Is that the right approach?
Diagonalization of a matrix as a linear map is different from a diagonalization of a matrix as a quadratic form. $P^{-1}AP$ is the linear map diagonalization. For a quadratic form, consider the coordinate change $x=P\hat x$, then $p(x)=x^TAx=\hat x^TP^TAP\hat x$, that is we have $P^TAP$ diagonalization instead. It can be done by completing the squares in $p(x)$ or $LDL$ decomposition.