Suppose, I have the following equation
$$x'(A-B)x = 0$$
where $x$ is an unknown vector and $A$ and $B$ are known matrices. Both $A$ and $B$ can be assumed to be positive semidefinite, but their difference is unlikely to be positive semi-definite.
(a) Is it possible to solve for $x$ (even if $x$ is not unique)?
(b) If vector $x$ is the solution of $x'Cx = 0$, where $C$ is a matrix, is there any special name for $x$?
We consider the equation in $x\in\mathbb{R}^n$: $x^TCx=0$ where $C$ is real symmetric. There is an invertible matrix $P$ s.t. $C=P^Tdiag(I_p,-I_q,0_{n-p-q})P$. Since $P$ is known, it suffices to find $y=Px$ s.t. $y^Tdiag(I_p,-I_q,0)y=0$. With respect to the previous block matrices, let $y=[y_1,y_2,y_3]^T$; we obtain $y_1^Ty_1-y_2^Ty_2=0$, that is $||y_1||=||y_2||$ -for the $l_2$ norm- That's all folks !
EDIT. Answer to the OP.
i) There is $Q\in O(n)$ s.t. $QCQ^T=diag((\lambda_i)_{i\leq p},(-\mu_j)_{j\in [[p+1,p+q]]},0_{n-p-q})$ where $\lambda_i>0,\mu_j>0$.
$QCQ^T=R^Tdiag(I_p,-I_q,0_{n-p-q})R$ where
$R=diag((\sqrt{\lambda_i})_i,(\sqrt{\mu_j})_j,I_{n-p-q})$. Then $P=RQ$ is invertible.
ii) If $C>0$, then $y=Px=0$. Since $P$ is invertible, $x=0$.