Solve for $x$ and $y$ :

Question

$$bx^3 = 10a^2bx + 3a^3y$$ $$ay^3 = 10ab^2y + 3b^3x$$

I tried putting $bx = p$ and $ay = q$ resulting into system of equations: $$p^3 = 10a^2b^2p + 3a^2b^2q$$ $$q^3 = 10a^2b^2q + 3a^2b^2p$$ Subtracting both I got $$p = q$$ OR $$p^2 + pq + q^2 = 7a^2b^2$$

Putting $p = q$ in cubic equations of $p$ and $q$, I got $p = 0$ or $p = ±√13ab$ implying , $bx = 0$ or $bx = ±√13ab$ i.e., $x = 0$ or $x = ±√13a$ and similarly $y = 0$ or $y = ±√13b$ but there are more values needed to be found. I am not getting how to proceed further. Please help in this direction.

Answer 1

Suppose that $a,b$ are nonzero. Taking the resultant of both polynomials in $x$ and $y$, with respect to $x$, we obtain $$ (13b^2 - y^2)(7b^2 - y^2)(3b + y)(3b - y)(b + y)(b - y)=0. $$ This yields all nontrivial solutions. Alternatively, you can eliminate $x$ from the second equation, provided that $b\neq 0$ and then substitute this in the first equation. Factoring gives then again the above solutions.

Answer 2

Subtracting both I got $\;\;p = q\;\;$ OR $\;\;p^2 + pq + q^2 = 7a^2b^2 \tag{1}$

Also, adding both you get $\;\;p = -q\;\;$ OR $\;\;p^2 - pq + q^2 = 13 a^2b^2 \tag{2}$

Subtracting $(1) - (2)$ gives $\;pq = -3a^2b^2\,$, then using $(1)$ gives $(p+q)^2=7a^2b^2 + pq = 4a^2b^2$. What's left is to solve the quadratics $\,t^2 \pm 2 a b\,t - 3a^2b^2 = 0\,$ for $\,p,q\,$.

---

[ EDIT ] $\;$ To eliminate the confusion around the special cases $p = \pm q$ here is the complete step-by-step breakdown.

Subtracting and adding the given equations gives the following two equations. That's without any simplifications, so these two equations are completely equivalent to the original system.

$$ \begin{align} (p-q)(p^2 + pq + q^2) &= 7a^2b^2 \,(p-q) \tag{1'} \\ (p+q)(p^2 - pq + q^2) &= 13 a^2b^2 \,(p+q) \tag{2'} \end{align} $$

At this point, we break down the problem into three cases.

I. $\,$ If $p-q = 0$ then $(1')$ is automatically satisfied, and $p = q$ can be determined by substituting into either of the original equations, as the OP has done in the posted question.

II. $\,$ If $p+q = 0$ then $(2')$ is automatically satisfied, and $p = -q$ can be determined the same way as in I.

III. $\,$ If $p-q \ne 0$ and $p+q \ne 0$ then the factors $p \pm q$ can be simplified in each equation, which leaves:

$$ \begin{align} p^2 + pq + q^2 &= 7a^2b^2 \tag{1} \\ p^2 - pq + q^2 &= 13 a^2b^2 \tag{2} \end{align} $$

Case III. is the case solved in my answer above. Together with the other two trivial cases I. and II., this provides the complete solution set to the original system.