I am trying to find the definite integral of $\frac{1}{a^2cos^2x+b^2sin^2x}$ over $0$ to $2\pi$.
I used the substitution $tan\ x=t$ but I am getting answer in terms of $tan^{-1}$, and that function over the given limits will be zero. But when I plotted the function to check for area, it is some positive value. I need help with such situations where substitutions and limits give zero answer.
Using your substitution, the indefinite integral seems to be
$$\frac1{ab}\arctan\left(\frac{b\tan x}a\right).$$
But this function has singularities at $x=\dfrac{2k+1}2\pi$, which are in fact jumps by $-\pi$. You can restore the true antiderivative by compensating the discontinuities, using
$$\frac1{ab}\left(\arctan\left(\frac{b\tan x}a\right)+\pi\left\lfloor\frac x\pi+\frac12\right\rfloor\right)$$ and finally the integral from $0$ to $2\pi$ is
$$\frac{2\pi}{ab}.$$