I was evaluating the following integral by means of contour integral
$$ \begin{aligned} \int_0^{\pi/2}\cos(a \theta)\cos^b(\theta)\,d \theta=\frac{\pi}{2^{b+1}(b+1)} \,\frac{1}{\operatorname{B}\left(\frac{a+b+2}{2},\frac{b-a+2}{2} \right)} \end{aligned} $$
For $a,b \in \operatorname{C}$, $\operatorname{Re}(a)>\operatorname{Re}(b)$, and $\operatorname{R}(b)>-1$.To this end I chose the following integrand
$$F(z)=z^{a-b-1}(1+z^2)^b=z^{a-b-1}(z+i)^b(z-i)^b$$
Which cleary for non integrers $a$ and $b$ has three branch points.
We want to compute the following integral along the contour $C_r$ below, where the dotted lines indicate the branch cuts. Since inside this contour the integrand has no singularities, by Cauchy´s teorem, the integral equals zero. Hence:
$$\oint_{C_r} z^{a-b-1}(1+z^2)^b \,dz=0$$
My question is regarding the integrals along the semicircles $A_1$,$A_2$ and $A_3$ that should vanish in the limit that $r$ goes to zero. I want to show that in fact those integrals vanish.
I will show the method I used for the contour $A_2$, the others should follow the same lines. I would like to know if this method is correct:
Choosing $z=re^{i \theta}$ the integral over $A_2$ becomes
$$ \begin{aligned} I_{A_2}&=\int_{\pi/2}^{-\pi/2} \left(re^{i \theta} \right)^{a-b-1}\left(1+r^2e^{2i \theta} \right)^b ire^{i \theta}\,d \theta\\ &=-i r^{a-b}\int_{-\pi/2}^{\pi/2} e^{i \theta(a-b)}e^{b \ln\left(1+r^2e^{2i \theta} \right)}\,d \theta\\ &=-i r^{a-b}\int_{-\pi/2}^{\pi/2} e^{i \theta(a-b)}e^{\frac{b}{2} \ln\left(\left(1+r^2\cos(2 \theta)\right)^2+r^4\sin(2 \theta)\right)+ib\arctan\left( \frac{r^2\sin(2 \theta)}{1+r^2 \cos(2 \theta)}\right) }\,d \theta\\ &\leq \Bigg|-i r^{a-b}\int_{-\pi/2}^{\pi/2} e^{i \theta(a-b)}e^{\frac{b}{2} \ln\left(\left(1+r^2\cos(2 \theta)\right)^2+r^4\sin(2 \theta)\right)+ib\arctan\left( \frac{r^2\sin(2 \theta)}{1+r^2 \cos(2 \theta)}\right) }\,d \theta \Bigg|\\ & \leq r^{a-b}\int_{-\pi/2}^{\pi/2} e^{\frac{b}{2} \ln\left(\left(1+r^2\cos(2 \theta)\right)^2+r^4\sin(2 \theta)\right) }\,d \theta \end{aligned} $$
taking $\lim_{r \rightarrow 0}$ in the last expression
$$ \begin{aligned} \lim_{r \rightarrow 0} \,I_{A_2}& \leq \lim_{r \rightarrow 0} r^{a-b}\int_{-\pi/2}^{\pi/2} e^{\frac{b}{2} \ln\left(\left(1+r^2\cos(2 \theta)\right)^2+r^4\sin(2 \theta)\right) }\,d \theta\\ & = \lim_{r \rightarrow 0} r^{a-b}\int_{-\pi/2}^{\pi/2} \lim_{r \rightarrow 0} e^{\frac{b}{2} \ln\left(\left(1+r^2\cos(2 \theta)\right)^2+r^4\sin(2 \theta)\right) }\,d \theta\\ & = \lim_{r \rightarrow 0} r^{a-b}\int_{-\pi/2}^{\pi/2} e^{\frac{b}{2} \ln\left(1\right) }\,d \theta\\ & = \lim_{r \rightarrow 0} r^{a-b}\int_{-\pi/2}^{\pi/2} \,d \theta\\ & = \pi \lim_{r \rightarrow 0} r^{a-b}\\ & \to 0 \end{aligned} $$
Is this procedure correct?
Outline for a fast estimation of the three integrals (the OP computations for $I_A$ are correct and similar but more complicated arguments can be used for the other two integrals, but imho it is simpler for the purpose at hand to use direct estimates).
One uses that $|\int_Cf(z)dz| \le l(C)\max_C|f(z)|$
So for the first integral (semicircle at zero) one has with the parametrization $z=re^{i\theta}$, the estimate $|z^{a-b}| \le Cr^{\Re a-\Re b-1}$ where $C=e^{\pi|\Im a -\Im b|/2}$ and $|(1+z^2)^b|\le C_1$ (with for example $C_1=2^{|\Re b|}e^{\pi |\Im b|}$ though one can do better of course - or simply use that $(1+z^2)^b$ is continuous on a small disc near zero so has a fixed undetermined bound $C_1$), while $l(C)=\pi r$ so the integral is at most $C_2r^{\Re a-\Re b} \to 0, r \to 0$ by the hypothesis
For the integral near $i$ (and same for the integral near $-i)$, one uses $z=i+re^{i\theta}$ with $0 < \theta_0 \le \pi$ and then $|(1+z^2)^b|=|(re^{i\theta})(2i+r{i\theta})|^b \le Cr^b$ with again $C$ easily estimated in terms of $\Re b, \Im b$ (something like $e^{2\pi|\Im b|}3^{|\Re b|}$ works and one can do better if needed), while $|z^{a-b}| \le C_1$ and $l(c) \le \pi r$ so the integral is at most $C_2r^{\Re b +1} \to 0, r \to 0$ by hypothesis