I am having difficulty reaching this integral result using complex analysis (Prob 4.1, Morse and Feshbach’s book: Methods of Theoretical Physics)
$$I=\int^{2\pi}_{0} \frac{\sin^{2}v}{a+b\cos v}dv= \frac{2\pi}{b^2}(a-\sqrt{a^{2}-b^{2}}); \ \ \ (a>b>0)$$
In my attempt I convert the integral to a close contour integral around the unit circle in the complex $z$ domain, with $z=1 e^{iv}$, giving the cosine and sine terms now in terms of $z$ and its conjugate $z^*$. Noting that $dv=dz/(iz)$ and that on the unit circle we also have $z z^{*}=1$, I get:
$$I=\oint \frac{(1/2)-z^{2}-z^{*2}}{2iz[2a+b(z+z^{*})]}dz$$
This gives two poles: one simple pole at center, which is included, and another simple pole whose real part is $\Re (z)<-1$ and hence outside the unit circle. Thus we have a residue at $z=0$ equal to $1/(8ia)$ and the integral finally gives $I=\pi/4a$. But this is not the stated answer in the problem.
What did I miss?
Your approach does not work as the residue theorem can only be applied to analytic functions. In particular, the function is only allowed to depend on $z$ but not on $z^*$. It is easy to fix though as on the unit circle $z^*= z^{-1}$. You then get $$ I=\oint \frac{2 - z^{2}- z^{-2}}{2iz[2a+b(z+z^{-1})]}dz =\oint \frac{2 z^2-z^{4}-1}{2i z^2[2a z+b(z^2+1)]}dz; $$ note, that there is a mistake in the numerator in your question.
You can observe that the simple poles are at $$ z_\pm= -\frac{a \pm \sqrt{a^2-b^2}}{b}\,. $$ Additionally, there is a double pole at $z=0$. The pole with $-$ is inside the unit circle and applying the residue theorem directly gives the result quoted.
Edit:
Here are a methods to calculate the residue at $z=z_-$ efficiently. As we know that the function has a simple pole at $z=z_-$, we can use the formula $$ \mathop{\rm Res}_{z=z_-} \frac{p(z)}{q(z)} = \lim_{z\to z_-} (z-z_-)\frac{p(z)}{q(z)} =\lim_{z\to z_-} \frac{p(z)}{q(z)/(z-z_-)} = \frac{p(z_-)}{q'(z_-)}\,. $$ In our case $p(z) = 2 z^2 -z^4-1 $ and $q(z) = 2i z^2(2az + b(z^2+1)) = 2 i b z^2(z-z_+)(z-z_-)$. Thus, we find ($z_- = 1/z_+$) $$ \mathop{\rm Res}_{z=z_-} \frac{p(z)}{q(z)} = \frac{2z_-^2 -z_-^4-1}{ 2i b z_-^2(z_--z_+)} = \frac{2 -z_-^2-z_+^2}{ 2i b (z_--z_+)} =- \frac{(z_--z_+)}{2 i b}= - \frac{\sqrt{a^2-b^2}}{i b^2}\,.$$
Moreover, we have to treat the double pole at $z=0$ to show that $$\mathop{\rm Res}_{z=0} \frac{p(z)}{q(z)} = -\frac{a}{i b^2}\,.$$
Together, we obtain the result quoted.
Comment:
For efficiency, I would always use a different mapping. In particular, you may observe that $$I = \mathop{\rm Re} \int_0^{2\pi}\, \frac{\frac12-\frac14 e^{2i v}}{a + b \cos v}\,dv\,.$$ Mapping this expression onto a contour integral, we get away with a much simpler function (in particular, without the double pole at $z=0$).