I need to show, that for any quadratic of the form $Ax^{2}+Bx+C$ with two distinct real roots and with $A<0$ I can find $a,b,k\in\mathbb{R} \left(a,k\neq0\right)$ S.T. $Ax^{2}+Bx+C=k^{2}-\left(ax+b\right)^2$
In the original context the question was about showing that $$I=\int\frac{1}{\sqrt{Ax^{2}+Bx+C}}dx=\int\frac{1}{\sqrt{k^{2}-\left(ax+b\right)^{2}}}dx$$
But unless I am missing something these two things are equivalent.
I tried to solve for $a,b,k$ and got $a=\sqrt{-A},\,\,\,b=-\frac{2B}{\sqrt{-A}},\,\,\,k^{2}=C+\frac{2B}{\sqrt{-A}}$. But then I need to show that $C+\frac{2B}{\sqrt{-A}}>0$ and this doesn’t seem to necessarily be true. Help?
First of all, yes the two things are equivalent, but the justification is not trivial. It requires a few words on the domain of the integral (between the roots of the polynomial), and arguments based on the positivity of the integral, on continuity, and equality of polynomials. The fact that the polynomial has two distinct real roots is crucial not only to be able to define such integral (with the hypothesis A<0) but also for your question.
On to the answer. You should redo your calculus. I think your value for b is wrong and therefore the one for $k^2$ as well. When you have the right values, you will have your answer. You will have to use the last hypothesis you didn't use: the one of the two roots. (Hint: what does it tell you in terms of discriminant?).