We have a quadratic form in the reals.
$$ q(x,y,z) = 2zx + 4yz - 2xy $$
($q$ is in the standard form of $V = R^3$)
We need to find: $(1)$ A base for $V$ in which $q$ is diagonalized. $(2)$ Find a subspace $W$ of $V$ with maximal dimension such that $q(w) \geq 0, \forall w \in W$
I am stuck in undertanding $(2)$.
$(1)$ Using elemtaric congruence we get:
Therefore our change of base matrix is:
Now we can show the base in which $q$ is diagonalaized:
NOW THE PROBLEM STARTS
They choose the subspace: $$ W = Sp \{(-1,0.5,0),(2,1,1)\} $$
Now they say: $$ \forall w \in W: [w]_{B} = (0, \lambda_1, \lambda_2) $$
Therefore: $$ q(w) = -1 \cdot 0^2 + \lambda_1^2 + 4 \lambda_2^2 \geq 0 $$
I dont understand
How did they know to take that subspace?
How $[w]_B = (0, \lambda_1, \lambda_2)$
How did they know?
Thank you.



Take a look at your diagonalization: in the new basis $B$, the quadratic form looks like $\langle -1,1,4\rangle$. You have one negative coefficient, and two positive ones. So if you want your quadratic form to be positive on a subspace, you should take the subspace spanned by the basis vectors corresponding to the positive coefficients.
In your case, these are the second and third basis vectors, so $(-1,0.5,0)$ and $(2,1,1)$. And of course, if you take a vector in this subspace and write it in the new basis $B$, by construction it has a first component equal to $0$.
EDIT: I see the new answer states that the diagonalization itself is incorrect, which is indeed the case. I did not try to check that before answering. This being said, I leave my answer since it does provide the method to find the correct subspace (once you get the right diagonanization of course).