Given the following Quadric $$F_4 := \{X \in R^3 | x_1x_2+x_1x_3+x_2x_3 =4\} $$
My task is find an orthogonal Matrix C and $d_1,d_2,d_3 \in R $ so that
$$F_4 = C*\{Y \in R^3 | d_1y_1^2 +d_2y_2^2+d_3y_3^2 =4\} $$
So far I've tried to write $ x_1x_2+x_1x_3+x_2x_3 =4 $ in the form $ X^t A X = 4 $
and got $$ A= \begin{matrix} 0 & 1 & 1 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \\ \end{matrix} $$
Using a theorem that the columns of $ C $ are the orthornormalized Eigenvectors from $ A $ I only got the Eigenvector (1,0,0), which can't be right.
Can someone point me to the right direction?
A symmetric matrix is guaranteed to have a basis of orthonormal eigenvectors. However, the matrix, $A$, that you got is not symmetric. In fact, it isn't even diagonalizable. This is why you only found one eigenvector.
Instead, try writing the quadratic as $$x_1x_2+x_1x_3+x_2x_3 = \dfrac{1}{2}x_1x_2+\dfrac{1}{2}x_2x_1+\dfrac{1}{2}x_1x_3+\dfrac{1}{2}x_3x_1+\dfrac{1}{2}x_2x_3+\dfrac{1}{2}x_3x_2.$$
Then, you can write the equation in the form $x^TAx = 4$ where $A$ is symmetric, and proceed from there.