Show that $$\int_{0}^{\infty}{\frac{\cos{x}}{x^\alpha}dx}=\mathrm{\Gamma}\left(1-a\right)\sin{\left(\frac{\mathrm{\pi\alpha}}{2}\right)}\ ,\ \ \ \ 0<\alpha<1$$
what would be the contour of this integration if I want to solve this integration using residue theorem
Assuming you wish to show $$\int_0^\infty \frac{\cos x}{x^\alpha} \, dx = \Gamma (1 - \alpha) \sin \left (\frac{\pi \alpha}{2} \right ), \qquad 0 < \alpha < 1,$$ what follows is a real method that can be used.
Let $$I = \int_0^\infty \frac{\cos x}{x^\alpha} \, dx.$$
By employing the following useful property for the Laplace transform: $$\int_0^\infty f(x) g(x) \, dx = \int_0^\infty \mathcal{L} \{f(x)\} (t) \cdot \mathcal{L}^{-1} \{g(x)\} (t) \, dt.$$ Noting that $$\mathcal{L} \{\cos x\}(t) = \frac{t}{1 + t^2},$$ and $$\mathcal{L}^{-1} \left \{\frac{1}{x^\alpha} \right \} (t)= \frac{1}{\Gamma (\alpha)} \mathcal{L}^{-1} \left \{\frac{\Gamma (\alpha)}{x^{(\alpha - 1) + 1}} \right \} (t) = \frac{t^{\alpha - 1}}{\Gamma (\alpha)},$$ then \begin{align} I &= \int_0^\infty \cos x \cdot \frac{1}{x^\alpha} \, dx\\ &= \int_0^\infty \mathcal{L} \{\cos x\} (t) \cdot \mathcal{L}^{-1} \left \{\frac{1}{x^\alpha} \right \} (t) \, dt\\ &= \frac{1}{\Gamma (\alpha)} \int_0^\infty \frac{t^\alpha}{1 + t^2} \, dt. \end{align} Setting $u = t^2$, one has \begin{align} I &= \frac{1}{2 \Gamma (\alpha)} \int_0^\infty \frac{u^{\frac{\alpha}{2} - \frac{1}{2}}}{1 + u} \, du\\ &= \frac{1}{2 \Gamma (\alpha)} \operatorname{B} \left (\frac{1}{2} - \frac{\alpha}{2}, \frac{1}{2} + \frac{\alpha}{2} \right ) \tag1\\ &= \frac{1}{2 \Gamma (\alpha)} \Gamma \left (\frac{1}{2} - \frac{\alpha}{2} \right ) \Gamma \left (\frac{1}{2} + \frac{\alpha}{2} \right ) \tag2\\ &= \frac{1}{2 \Gamma (\alpha)} \Gamma \left [1 - \left (\frac{1}{2} + \frac{\alpha}{2} \right ) \right ] \Gamma \left (\frac{1}{2} + \frac{\alpha}{2} \right )\\ &= \frac{1}{2 \Gamma (\alpha)} \frac{\pi}{\sin \left (\frac{\pi}{2} - \frac{\pi \alpha}{2} \right )} \tag3\\ &= \frac{1}{2 \Gamma (\alpha)} \frac{\pi}{\cos \left (\frac{\pi \alpha}{2} \right )}\\ &= \frac{\Gamma (1 - \alpha) \sin (\pi \alpha)}{2 \pi} \cdot \frac{\pi}{\cos \left (\frac{\pi \alpha}{2} \right )} \tag4\\ &=\frac{\Gamma (1 - \alpha) \sin \left (\frac{\pi \alpha}{2} \right ) \cos \left (\frac{\pi \alpha}{2} \right )}{\cos \left (\frac{\pi \alpha}{2} \right )}\tag5\\ &= \Gamma (1 - \alpha) \sin \left (\frac{\pi \alpha}{2} \right ), \end{align} as required to show.
Explanation
(1) Using $\operatorname{B} (x,y) = \displaystyle{\int_0^\infty \frac{t^{x - 1}}{(1 + t)^{x + y}} \, dt}$.
(2) Using $\operatorname{B}(x,y) = \dfrac{\Gamma (x) \Gamma (y)}{\Gamma (x + y)}$.
(3) Using the reflexion formula for the gamma function: $\Gamma (1 - z) \Gamma (z) = \dfrac{\pi}{\sin (\pi z)}$.
(4) Again using the reflexion formula for the gamma function.
(5) Using the double angle formula for sine.