Let $q:\mathbb{R^3}\to\mathbb{R}$ such that $$q(x,y,z)=2x^2+3y^2+4xy-2xz.$$
I have to determine rank and signature of $q$, and so far it should be fine: I got $\operatorname{rk}(q)=3$ and $\operatorname{sgn}(q)= (2,1)$. The second question is: let $V=\langle(1,-1,0),(1,0,0)\rangle$, vector subspace of $\mathbb{R^3}$, determine rank, signature and the set of isotropy vectors of the restriction of $q$ to $V$.
Here I'm a bit confused, also because we've a two dimensional subspace of a three dimensional space. Could you show me how to work this example step by step (also showing how to calculate the matrix associated with the restriction of $q$)? Could you also add some general remarks on "how to work" with restriction of quadratic forms?
That plane $V$ is described by $z=0$, hence your restriction is just
$$q(x,y,0)=2x^2+3y^2+4xy$$
This is then
$$q'(x,y)=2x^2+3y^2-4xy$$
which is represented by the matrix
$$\begin{pmatrix} 2 & 2 \\ 2 & 3\end{pmatrix}$$
The eigenvalues are ${5\over 2}\pm {\sqrt{17}\over 2}$, which are both positive.
so the new signature is $(2,0)$, in particular it's positive definite, so that the form is anisotropic and the rank is $2$.
In general the idea is that in an ambient space, every linear space is the intersection of hyperplanes. So say you're in $\Bbb R^m$ and you want the restriction of the form to the subspace spanned by the (linearly-independent) $v_1,\ldots, v_k$. Then we know any $k$-dimensional subspace can be determined as the zero-set of $m-k$ linear forms. In our specific example, the linear form was $0x+0y+z=0$.
I used that to replace the variable $z$ by a function of $x$ and $y$, i.e. $z=0-0x-0y=0$ by moving the $x$ and $y$ data to the other side.
In a general setting you'd pick your preferred variables, and solve for the other ones in terms of those variables using the linear forms, then make substitutions into the form to reduce the number of variables.
Example
say we did your original form, but the space was spanned by $\langle 1, -1, 0\rangle$ and $\langle 1, 0, -1\rangle$ so that the linear form describing the space is $x+y+z=0$, then if I like $x$ and $y$ best I do $z=-(x+y)$ then I go back to the original form and plug in
$$q(x,y,-(x+y))=2x^2+3y^2+4xy+2x(x+y)= 4x^2+3y^2+6xy$$
so $q'(x,y)$ is represented by the matrix
$$\begin{pmatrix} 4 & 3 \\ 3 & 3\end{pmatrix}$$
If, instead, we wanted to restrict to the $1$-dimensional linear subspace spanned by $\langle 1, 1,3\rangle$ we note this is the zero set of the two linear forms: $x-y$ and $x-z$. This means $x=y$ and $3x=z$ so that we have the new form
$$q(x,x,3x)=2x^2+3x^2+4x^2-6x^2=3x^2$$
so $q'(x)=3x^2$ is represented by the matrix $3$.