Let $b_1, b_2,b_3$ be real constants, how to solve the following non-linear system? \begin{align} (-3x^2 + y^2 -6xy) + b_1 &= 0 \qquad\text{(L1)}\\ (\phantom{-} x^2 -3 y^2 -6xy) + b_2 &= 0 \qquad\text{(L2)}\\ (\phantom{.}3x^2 + 3 y^2 -2xy) + b_3 &= 0 \qquad\text{(L3)} \end{align}
I try (L1)-(L2) which yields: $y^2 = x^2 + \frac{b2-b1}{4}$.
Then, (L2)-3*(L3) leads to this intermediate results: \begin{align} x^2 &= \frac{3b_1-2b_2-12b_3}{20}\\ y^2 &= \frac{-2b_1+3b_2-12b_3}{20} \end{align}
At this point, can I conclude the following statement? \begin{align} |x| &= \sqrt{\frac{3b_1-2b_2-12b_3}{20}}\\ |y| &= \sqrt{\frac{-2b_1+3b_2-12b_3}{20}} \end{align} What if $(3b_1-2b_2-12b_3) <0$ or $(-2b_1+3b_2-12b_3)<0$? Does it mean that the system has no solution?
If I plug the previous results of $x^2$ and $y^2$ in (L1), (L2) and (L3), then I obtain: \begin{align} xy &= -\frac{1}{40}(3b_1 + 3b_2 + 8b_3)\\ xy &= -\frac{1}{40}(3b_1 + 3b_2 - 52b_3)\\ \end{align} which is valid only if $b_3 = 0$. In this case $xy = -\frac{3}{40}(b_1 + b_2)$. Can I conclude something about the sign of $x$ and $y$?
Since you have three equations and two unknowns, I should define $z=xy$ to make $$\begin{align} (-3x^2 + y^2 -6z) + a &= 0 \tag 1\\ (\phantom{-} x^2 -3 y^2 -6z) + b &= 0 \tag 2\\ (\phantom{.}3x^2 + 3 y^2 -2z) + c &= 0 \tag 3 \end{align}$$
So, four solutions $$x=\pm\frac{\sqrt{3 a-2 b-3 c}}{2 \sqrt{5}}\qquad y=\pm \frac{\sqrt{3b-2 a-3 c}}{2 \sqrt{5}}\qquad z=\frac{3 a+3 b+2 c}{40}$$
So, as you noticed, if we are in the real domain, this gives already two conditions $3a-2b-3c \geq 0$ and $3b-2a-3c>0$. If these ared fulfilled, then $z=xy$ gives a relation between $a$, $b$ and $c$, but no more.