How can I find the positive integer solutions to $x$ and $y$, given that $n$, $a$, $b$ and $c$ are all positive integers, in an equation of the form:
$$n = ax^2 + by^2 - cxy.$$
Specifically, I want to find the positive integer solutions to the following equation, given $n$:
$$n=3 x^2+20 y^2-16 x y.$$
$$ n = (3x - 10y)(x-2y) {}{} $$
Alright, changing variables with $$ u = x - 4 y, \; v = x - 3 y $$ so that $$ x = 4 v - 3 u, \; y = v - u, $$ we have $$ 3 x^2 - 16 x y + 20 y^2 = -u^2 + 4 v^2. $$
So, factoring $n,$ we find all possible ways to write $$ n = -u^2 + 4 v^2 = (2 v + u )(2 v - u) $$ as an even square minus an odd square, definitely including both $u,-u$ for each success, also both $v,-v.$ Then, for each success (finitely many) we switch back to the original variables with $ x = 4 v - 3 u, \; y = v - u, $ and choose the solutions that include your conditions on positivity, whatever you might have meant by that.
Note that there are no solutions if $n \equiv 1,2 \pmod 4.$ When $n<0,$ there are no solutions when $ |n| \equiv 2,3 \pmod 4. $
EDIT, Wednesday morning, before Gerry wakes up in Australia, another completely cosmetic change: switch to $-n$ and find all solutions to $$ -n = s \, (s + 4 t) $$ which is just to find ALL ways of writing $$ -n = FG $$ such that $F$ and $G$ differ by a multiple of $4.$ Then use $$ x = -3s - 2 t, \; y = -s-t, $$ from $$ s = -x+2y, \; t = x-3y. $$ So There.
Well, if $n=0$ I guess there really are infinitely many solutions. But, as soon as $n \neq 0,$ we get finitely many solutions because $$ |s| \leq |n|, \; \; |s+4t| \leq |n|. $$