The Diophantine equation of the form a$x^2$ – b$y^2$ = $c^2$ with ab is not a perfect square in Z has infinitely solutions in N, provided by a particular non-trivial solution in set of N.
I have racked my brains trying to think why ab not a perfect square should invalidate the proof, but can't think why. I have many books on number theory, but none have an equation like this.
If any one can help me in this aspect...I am so thankful to them.
On
The choice of having $c^2$ on the right hand side is irrelevant, any nonzero number gives the same conclusions.
Let $$ g = \gcd(a,b). $$ By unique factorization, with $$ \gcd \left( \frac{a}{g}, \frac{b}{g} \right) = 1, $$ the product being a square gives $$ a = g \alpha^2, \; \; b = g \beta^2, $$ and let us take $g, \alpha, \beta > 0.$ So your equation becomes $$ c^2 = a x^2 - b y^2 = g (\alpha^2 x^2 - \beta^2 y^2) = g (\alpha x - \beta y) (\alpha x + \beta y). $$ Now, either $\alpha x, \; \beta y$ have the same sign or opposite. With $c \neq 0$ we get get one factor at least $1$ in absolute value, so then $$ |\alpha x| + | \beta y| \leq \frac{c^2}{g}, $$ so $$ | x| \leq \frac{c^2}{g \alpha} $$ and $$ | y| \leq \frac{c^2}{g \beta}, $$ giving finiteness of the set of solutions.
MEANWHILE, if $ab$ is not a perfect square, there are infinitely many solutions to the Pell equation $$ u^2 - a b v^2 = 1. $$ This makes infinitely many different solutions if there are any, because $$ a (ux + b v y)^2 - b (avx + uy)^2 = a x^2 - b y^2. $$
I'm not sure what you are asking. If you are asking for an example where $ab$ is a perfect square and the equation doesn't have infinitely many solutions, perhaps the simplest example is that with $a=b=1$.