With the following equation in integers: $$x^2 - xy + ay^2 = b \tag{1}$$ I would like to understand what values of $a$ and $b$ would let us conclude that if any solution exists, it must have $y=0$.
If I choose some values for $a$ and $b$, then plug in values for $x$ and $y$, I can see that the minimum value of $x^2 - xy + ay^2$ with a non-zero $y$ lets me conclude that if a solution exists, it must have $y=0$. But this plug-and-chug method is too crude to extract any nice relationships on $a$ and $b$ which lead to this conclusion.
By contrast, with a simpler equation, $$x^2 + ay^2 = b \tag{2}$$ I can see that $y=0$ if $a>b>0$ because regardless of the value of $x$, increasing $|y|$ increases $x^2 + ay^2$, and similarly, regardless of the value of $y$, increasing $|x|$ increases $x^2 + ay^2$, so it is easy to see that the smallest possible value of $x^2 + ay^2$ with a non-zero $y$ is $a$. So $a>b>0$ leads to the requirement that $y=0$.
What is giving me trouble is the $-xy$ term. I don't know how to reason about $x^2 -xy + ay^2$, because sometimes increasing $y$ would actually decrease the total. If I plot it like a surface I see it curves up if $a>0$, so it feels like I should still be able to say something similar, but I'm not sure how to show it rigorously.
For what ranges of $a$ and $b$ can I conclude that if there is a solution to eq1, that it must have $y=0$?
First consider this in the real numbers, and we'll restrict to integers later.
Let $f(x,y) = x^2-xy+ay^2$. Given a value for $y$, this just becomes a parabola in $x$ with a minimum at $\partial_x f = 2x - y = 0$. So for a given value for $y$ the minimum value is $$f\left(\frac{y}{2},y\right) = \frac{1}{4}y^2 -\frac{1}{2}y^2 + ay^2 = \left(a - \frac{1}{4}\right)y^2.$$
Now if $y$ is restricted to non-zero integers, the minimum is $(a - \frac{1}{4})$. Thus $y$ can't be a non-zero integer if $a > b + \frac{1}{4} > b$.
Then if all values are integers, we have:
$$x^2-xy+ay^2=b, \quad a > b \quad \longrightarrow \quad y=0.$$