Please help me to solve this question or introduce references that help me:
Let $f(x,y) = ax^2 + bxy + cy^2$ be a reduced positive definite form. Suppose that $\gcd(x, y) = 1$ and that $f(x, y) ≤ a + |b| + c$. Show that $f(x,y)$ must be one of the numbers $a, c, a - |b| + c$ or $a + |b| + c$.
I suppose the main thing is that $$ f \geq \left( \frac{4ac - b^2}{4c} \right) x^2, $$ $$ f \geq \left( \frac{4ac - b^2}{4a} \right) y^2. $$ These just come from completing the square.
Reduced means $$ |b| \leq a \leq c $$ with some additional conditions in case some things are equal. For example, $ac \geq b^2.$ Thus $4ac-b^2 \geq 3ac.$ Thus $$ f \geq \left( \frac{3a}{4} \right) x^2, $$ $$ f \geq \left( \frac{3c }{4} \right) y^2. $$
Now, suppose $|y|\geq 2,$ so that $y^2 \geq 4$ and $f \geq 3c.$ With the condition $f \leq a + |b| + c$ and reduction this gives $y = \pm 2,$ $a = |b| = c$ and so $a=|b| = c = 1$ because $\gcd(a,b,c) = 1.$ Take $y=2$ and continue with $x^2 + 2x + 4.$
Next, if we do not have $a=|b| = c = 1,$ we have $y = \pm 1$ or $y=0.$ Continue with either $ax^2 + b x + c$ and $y=1$ or $a x^2$ with $y=0.$