I am looking to count number of solutions of diophantine equation $k_1a^2+k_2ab+k_3b^2-k_4c^2=0$.
such that $ 1 \le a, b, c \le N$ and $gcd(a, b) = 1$ and $k_1,k_2,k_3,k_4$ are positive constant integers
We don't need all solutions set, we just need to count the number of solutions.
I have an approach in mind which uses Meissel Lehmer's algorithm and solves in $O(N^{3/4}log^2 N)$ but
My guess is this can be solved in something of the order of $O(N^{1/2})$ or similar using different approach.
Let's start from the generalized equation
$$(2x+ay)^2 + (4b-a^2)y^2 = 4cz^2$$
where $a = k_2/k_1, b=k_3/k_1, c=k_4/k_1$, and I've simply used $x,y,z$ rather than $a,b,c$ to make things easier to read.
One of the more interesting solution methods is to note that, because all the terms are degree $2$, any multiple of a solution is also a solution. Which means that we can divide the equation above through by $y^2$ to get:
$$4b-a^2 = 4c \left( \frac z y \right)^2 - \left[ 2 \left( \frac x y \right) + a \right]^2$$
We substitute $Y = z/y, X = x/y$, and we can graph this in the plane as
$$4cY^2 - (2X+a)^2 = 4b-a^2$$ $$\implies cY^2 - X^2 - aX + b = 0$$
This is a hyperbola; whether it opens up/down or left/right depends on the $k_i$. Now the kicker is this: every rational point $(X,Y)$ on this hyperbola corresponds to a family of solutions of the original equation, and this is, I believe, a bijection. The trick, of course, is finding an initial rational point $(s,t)$--though if there is one rational point, there are infinitely many.
Every other rational point can be found by generating a line of rational slope $p/q$ going through the initial rational point, and finding the intersection of the hyperbola and the line
$$Y = \frac p q X + \frac{qt-ps}{q}$$
And if you can do that, you can find a parametrization in $p,q$ for every solution. You can check my algebra--
(I probably messed something up!)I did mess up my calculations and have edited to fix, twice, but am pretty certain now--but as a parametrization I get:\begin{cases} x = \frac{n}{s}(2cpqst+bq^2-cp^2s^2-cq^2t^2) \\ y = n(q^2-cp^2) \\ z = \frac{n}{s}(cp^2st+q^2st+bpq-cpqt^2-pqs^2) \end{cases}
Where $n$ is a rational coefficient that gets you integers; YMMV. This is assuming you can find a rational point $(s,t)$ with $s \neq 0$, of course. I have no idea if this helps with counting the possible solutions below a certain $N$ though. Also note that, like $k_i$ earlier, $p,q \in \Bbb{N}$. Note there is no guarantee of $(x,y)=1$ here, but of course you can easily find that solution.
(Edited twice for algebra)