I'm trying to find an (smallest) integer solution of the equation $$F(x_0, x_1, y_0, y_1) = (a\cdot x_0 +b\cdot x_1) \cdot (a\cdot y_0 + b\cdot y_1) - c,$$
where $a,b,c$ are known integers and ($x_0, x_1, y_0, y_1)$ are variables.
I already know that this equation has the smallest integer solution $(X_0,X_1, Y_0,Y_1)$ (with respect to Euclidean norm).
Question: Is there an algorithm to find the solution?
I will assume that $c\neq 0$, otherwise the problem is rather trivial. Clearly, if there is a solution, $ax_0+bx_1$ is a divisor $d$ of $c$, and in this case the other factor is $c/d$.
Hence you have to solve the two equations $ax_0+bx_1=d$ and $ay_0+by_1=c/d$ for each divisor $d$.
But the equation $au+bv=m$ is easily solved (where $a,b,m$ are given integers, and $m$ is not zero)
It has a solution if and only if $\delta=gcd(a,b)$ divides $m$.
If so, dividing by this gcd , it remains to solve $a'u+b'v=m'$ where $a'=a/\delta, b'=b\delta, m'=m/\delta$. Note that $a',b'$ are coprime.
Now use Euclid algorithm to find $\alpha,\beta$ such that $a'\alpha+b'\beta=1$, and set $u_0=m\alpha, v_0=m\beta$. Then $a'u+b'v=m=a'u_0+b'v_0$. Using the fact that $a',b'$ are coprime, we see that $u=u_0+kb',v=v_0-ka', k\in\mathbb{Z}$.