Evaluation of $\displaystyle \int\frac{1}{(a^2-\tan^2 x)\sqrt{b^2-\tan^2 x}}dx\;,$ Where $a>b$
$\bf{My\; Try::}$ Let $b^2-\tan^2 x= t^2\;,$ Then $-2\tan x \sec^2 xdx = 2tdt$
So integral convert into $\displaystyle I = -\int\frac{1}{(a^2+b^2-t^2)\cdot t}\times \frac{t}{(1+b^2-t^2)}\times \frac{1}{\sqrt{b^2-t^2}}dt$
Now how can i solve it, Help required, Thanks
On
Starting from $$I= \int\frac{dx}{(a^2-\tan^2 x)\sqrt{b^2-\tan^2 x}}$$ and using, as you did, $$b^2-\tan^2 x= t^2\implies x=-\tan ^{-1}\left(\sqrt{b^2-t^2}\right)\implies dx=\frac{t}{\sqrt{b^2-t^2} \left(b^2-t^2+1\right)}$$ $$I=\int\frac{dx}{\sqrt{b^2-t^2} \left(b^2-t^2+1\right) \left(a^2-b^2+t^2\right)}$$ The integrand can be decomposed as $$\frac{1}{a^2 \sqrt{b^2-t^2}}+\frac{\sqrt{b^2-t^2}}{\left(-a^2-1\right) \left(b^2-t^2+1\right)}-\frac{\sqrt{b^2-t^2}}{a^2 \left(a^2+1\right) \left(-a^2+b^2-t^2\right)}$$ leading, as given in comments, to $$a(a^2+1)\,I=\frac{\tan ^{-1}\left(\frac{a t}{\sqrt{a^2-b^2} \sqrt{b^2-t^2}}\right)}{\sqrt{a^2-b^2}}+\frac{a \tan ^{-1}\left(\frac{t}{\sqrt{b^2+1} \sqrt{b^2-t^2}}\right)}{\sqrt{b^2+1}}$$
Edit
Continuing from Rohan's answer, partial fraction decomposition leads, for the integrand, to $$\frac 1 {a^2+1}\left(\frac 1 {a^2+(a^2-b^2)w^2}+\frac 1{1+(b^2+1)w^2} \right)$$ which is much easier to integrate. This would lead to $$(a^2+1) \,I=\frac{\tan ^{-1}\left(\frac{ \sqrt{a^2-b^2}}{a}w\right)}{a \sqrt{a^2-b^2}}+\frac{\tan ^{-1}\left(\sqrt{b^2+1} w\right)}{\sqrt{b^2+1}}$$
I have an idea.
Substitute $u=\tan x $ to get $$I =\int \frac {1}{\sqrt {b^2-u^2}(u^2+1)(a^2-u^2)} \mathrm {d}u$$ Now substituting $u = b\sin v $, we get, $$I = \int \frac {1}{(b^2\sin ^2 v+1)(a^2-b^2\sin^2 v)}\mathrm {d}v $$ We then substitute for $w =\tan v $, to get, $$I =\int \frac {w^2+1}{((a^2-b^2)w^2+a^2)((b^2+1)w^2+1)} \mathrm {d}w $$ Now we do a partial fraction decomposition which I am unable to get right. Hope you can take it from here.