I'd like to solve $\displaystyle \int_0^\infty \frac{x^\alpha}{x(x+1)}dx,$ where $\alpha \in (0,1).$ The answer is $\frac{\pi}{\sin (\alpha \pi)}.$
Typically, to solve it, we use contour integral and residue theorem as follows. Let $C=C_{r}^R-I_{r,R}^- -C_r+I_{r,R}^+$ denote the simple closed positively oriented contour, where $-C_r$ and $C_{r}^R$ are the portions of the circles $C_r(0)$ and $C_R(0)$, respectively, and $-I_{r,R}^-$ and $I_{r,R}^+$ the horizontal segments joining them. We select a small value of $r$ and a large value of $R$ so that the nonzero poles $-1$ of $\displaystyle f(z):=\frac{z^\alpha}{z (z+1)}$ lie inside $C.$ We use the branch of $z^\alpha$ corresponding to the branch of the logarithm $\log_0$ as follows:
$\displaystyle z^\alpha=e^{\alpha \log_0 (z)} =|z|^\alpha e^{i \alpha\theta}$ for $z=re^{i\theta}\neq 0$ and $\theta \in (0,2\pi]$
Then $\displaystyle \int_C f(z)dz=\int_{C_{r}^R}f(z)dz-\int_{I_{r,R}^-}f(z)dz -\int_{C_r}f(z)dz+\int_{I_{r,R}^+}f(z)dz.$
Using the residue theorem, \begin{equation}\label{8-29} \int_C f(z)dz=2\pi i \textrm{Res}[f,-1]. \end{equation} On the other hand, $\displaystyle \lim_{r\to 0^+}\int_{C_r^R}f(z)dz=\int_{C_R^+(0)}f(z)dz$, and it can be shown that $\displaystyle\lim_{r \to 0^+} \int_{C_r}f(z)=\lim_{R\to \infty}\int_{C_R^+(0)}f(z)dz=0,$ by M-L inequlity.
My question arises in the next part (most textbooks don't explain it in detail):
Because of the branch we chose for $z^\alpha,$ $\displaystyle \lim_{r\to 0^+} \int_{I_{r,R}^+}f(z)dz=\int_0^R\frac{x^\alpha}{x(x+1)}$ and $\displaystyle\lim_{r\to 0^+} \int_{I_{r,R}^-}f(z)dz=\int_0^R\frac{x^\alpha e^{i\alpha 2\pi}}{x(x+1)}$. Note that since $Q$ has a zero of order at most $1$ at the origin, the above two integrals converge.
I understand that the integrand $f(z)$ on the upper horizontal line approaches $\frac{x^\alpha}{x(x+1)}$, where $x$ is a real number. Similarly, $f(z)$ on the lower horizontal line approaches $\frac{x^\alpha e^{i\alpha 2\pi}}{x(x+1)}$.
I'd like to prove $\displaystyle \lim_{r\to 0^+} \int_{I_{r,R}^+}f(z)dz=\int_0^R\frac{x^\alpha P(x)}{Q(x)}$ in detail.
What theorem do I use for this equality. Do I use Lebesgue dominated convergence theorem?
I appreciate if you give any comments about it. Thanks in advance.
It depends on your definition of an (improper) integral like $\int_0^R\frac{dx}{\sqrt x}$. If you see it as a Riemann integral, then most likely you've defined it precisely as the limit of $\int_r^R\frac{dx}{\sqrt x}$ for $r\to 0^+$, and there's nothing to prove. (By the way, this is also how you would calculate it.)
If you've seen Lebesgue integrability, then there's no need to take limits to define $\int_0^R\frac{dx}{\sqrt x}$. In that case, to prove that it equals $\lim_{r\to0^+}\int_r^R\frac{dx}{\sqrt x}$, you can use dominated convergence indeed. Note that it suffices to find the limit for all sequences $r_n\to0^+$. By dominated convergence (and the fact that Riemann and Lebesgue coincide on closed intervals, say), $$\begin{align*} \lim_{r_n\to0^+}\int_{r_n}^R\frac{dx}{\sqrt x} &=\lim_{r_n\to0^+}\int_{(0,R]}\frac{dx}{\sqrt x}\cdot{\bf1}_{[r_n,R]}\\ &=\int_{(0,R]}\lim_{r_n\to0^+}\frac{dx}{\sqrt x}\cdot{\bf1}_{[r_n,R]} &=\int_{(0,R]}\frac{dx}{\sqrt x} \end{align*}$$ because the last integrand bounds the others and is Lebesgue integrable.
In your case, with $x^{-\beta}g$ instead of $x^{-1/2}$, with $\beta\in(0,1)$ and $g$ continuous in a neighborhood of $0$ (i.e. a rational function), the same arguments work: with the Riemann definition there's nothing to prove; with the Lebesgue definition, use DC and the fact that $g$ is bounded in a neighborhood of $0$ to justify it.