I want to solve a matrix $x^\top\!Mx=0$ as follows, $$ x^\top\begin{bmatrix} O_{3\times3} & A \\ B & C \end{bmatrix}x=0 $$
In addition, there is a property of matrix $A$ and $B$, i.e. $A^\top\!B=I$.
I want to know a better method to solve $M$ using $A^\top\!B=I$.
Assume A is a $3\times k$ matrix and B is a $k\times 3$ matrix.
Since $A^TB=I$, A and B has same number of rows so that both A and B are $3 \times 3$ square matrices too. u' Let $x^T=(u^T, v^T)$ we have $u^TAv+v^TBu+v^TCv=0$ or $u^T(A+B)v+v^TCv=0$ since $v^TBu=u^TBv$ (it is a $1\times 1$ matrix).
So for any vector v, as soon as $(A+B)v \ne 0$, we could find infinite number of u so that $u^T (A+B)v =-v^TCv$.
If $v^TCv=0$, we should choose any u which is perpendicular to (A+B)v. If $v^TCv \ne 0$, choose any w which is not perpendicular to (A+B)v, calculate $s= w^T (A+B)v \ne 0$, let $u=\frac{-v^TCv}{s}w$