Let $a,b,c$ be real numbers with $a > 0$. Suppose that $d = b^2 - 4ac > 0$. Is there a constant $e$ such that there exist integers $x,y$ not both $0$ and that $\left|ax^2 + bxy + cy^2\right|\le e\sqrt d$ ?
That question is a modified form of Problem 12 in Section 6.4, "The Geometry of Numbers," in I. Niven, H. S. Zuckerman, H. L. Montgomery, An Introduction to the Theory of Numbers, 5th ed., Wiley (New York), 1991.
The solution to Problem 12 was relatively easy because the region is that of an ellipse, which is convex and has an easy-to-calculate area, so all we had to do was use a modified version of Minkowski's convex body theorem.
The region of the question posed above, however, is not convex and has a weird shape; part of its boundary is hyperbolic.
$\begin{array}\\ f(x, y) &=ax^2+bxy+cy^2\\ &=ay^2((x/y)^2+(b/a)(x/y)+(c/a))\\ &=ay^2((x/y)-r)((x/y)-s)\\ \end{array} $
where $r$ and $s$ are the roots of $x^2+(b/a)x+(c/a) =0$.
These roots are
$\begin{array}\\ r, s &=\dfrac{-b/a\pm\sqrt{b^2/a^2-4c/a}}{2}\\ &=\dfrac{-b\pm\sqrt{b^2-4ac}}{2a}\\ &=\dfrac{-b\pm\sqrt{d}}{2a}\\ &= u \pm v \qquad\text{where } u = -\dfrac{b}{2a}, v = \dfrac{\sqrt{d}}{2a}\\ \end{array} $
Since $d > 0$, $r$ and $s$ are real.
Therefore $f(x, y) =ax^2+bxy+cy^2 =ay^2((x/y)-r)((x/y)-s) =a(x-ry)(x-sy) $. So we want $|a(x-ry)(x-sy)| \le e\sqrt{d} $.
Choosing $r = u+v $ and $s = u-v $,
$\begin{array}\\ a(x-ry)(x-sy) &=a(x-(u+v)y)(x-(u-v)y)\\ &=a(x-uy-vy)(x-uy+vy)\\ &=a((x-uy)^2-(vy)^2)\\ \end{array} $
so we want $e\sqrt{d} \ge a|(x-uy)^2-(vy)^2| $.
For any integer $y$, we can choose an integer $x$ such that $|x-uy| \lt 1$.
If $|y| = 1$ then $|(x-uy)^2-(vy)^2| \lt 1+v^2 $ so $e\sqrt{d} \ge a(1+v^2) $ will certainly do.
This is $e\sqrt{d} \ge a(1+\frac{d}{4a^2}) $ or $e \ge \dfrac{a}{\sqrt{d}}(1+\dfrac{d}{4a^2}) = \dfrac{a}{\sqrt{d}}+\dfrac{\sqrt{d}}{4a} $.