I would like this integral: $$\frac{1}{4}\int \frac{dx}{(kx^{2}+ax+b)^{2}}$$ I used substitution $x+ \frac{a}{2k} = \frac{\sqrt{4kb-a^{2}}}{2k} \tan \theta$.
This will reduce the denominator to $$ \left(\left(\frac{\sqrt{4kb-a^{2}}}{2k}\tan\theta \right)^{2} + \frac{4kb-a^{2}}{4k^{2}}\right)^{2} = \left(\frac{4kb-a^{2}}{4k^{2}}\right)^{2} \sec^{4}\theta ,$$
and integral is reduced to $$\frac{1}{4k^{2}}\frac{16k^{4}}{(4kb-a^{2})^{\frac{3}{2}}2k} \int \cos^{2} \theta d\theta = \frac{2k}{(4kb-a^{2})^{\frac{3}{2}}}\left(\frac{\theta}{2}+\frac{1}{4}\sin(2\theta)+c\right)$$ $$= \frac{2k}{(4kb-a^{2})^{\frac{3}{2}}} \left(\frac{\arctan\left(\frac{2k(x+\frac{a}{2k})}{\sqrt{4kb-a^{2}}}\right)}{2}+\frac{\sin(2 \arctan\left(\frac{2k(x+\frac{a}{2k})}{\sqrt{4kb-a^{2}}}\right)}{4}\right).$$
Is there a simpler solution?
The method is correct. You just made some computational mistakes.
You got the substitution and denominator correctly.
The integral is reduced to $\frac{1}{2\sqrt{4bk-a^2}}\int \cos^2{\theta} d\theta$. You also did the integral correctly.