I'm looking for several ways to prove that $$\int_{0}^{\infty }\sin(x)x^mdx=\cos\left(\frac{\pi m}{2}\right)\Gamma (m+1)$$
for $-2< Re(m)< 0$
I'm looking for several ways to prove that $$\int_{0}^{\infty }\sin(x)x^mdx=\cos\left(\frac{\pi m}{2}\right)\Gamma (m+1)$$
for $-2< Re(m)< 0$
On
Use the identities $$ \int\limits_{0}^{\infty}\mathrm dx\,x^{m}e^{-ax}= \frac{\Gamma (m+1)}{a^{m+1}}$$ Set $i= \exp(\pi i)$, so $\operatorname{Re}(\exp(\pi s))=\cos(\pi s) $ and you get your result.
On
Step1 Find that $$I_1(m)=\int_0^{\infty} \sin(x)x^m dx=\cos\frac{\pi m}{2}\Gamma(m+1),$$ $$I_2(m)=\int_0^{\infty}\cos(x)x^mdx=-\sin\frac{\pi m}{2}\Gamma(m+1),$$ for $-1<m<0$.
This can be done by coutour integration using quarter-circle contour.
Step2 The first integral $I_1(m)$ defines an analytic function on $-2<\Re m<0$.
This can be done by @sos440 's method of showing integrability.
Step3: Completion The identity for $I_1(m)$ should hold for all $m$ with $-2<\Re m<0$ by analytic continuation.
Here is a fake proof: Let $x = \sqrt{t}$. Then
$$ \int_{0}^{\infty} x^{m} \sin x \, dx = \frac{1}{2} \int_{0}^{\infty} t^{m/2} \frac{\sin \sqrt{t}}{\sqrt{t}} \, dt. $$
Since
$$ \frac{\sin \sqrt{t}}{\sqrt{t}} = \sum_{n=0}^{\infty} \frac{\phi(n)}{n!} (-t)^{n} \quad \text{for} \quad \phi(n) = \frac{n!}{(2n+1)!} = \frac{\Gamma(n+1)}{\Gamma(2n+2)}, $$
Ramanujan's master theorem gives
\begin{align*} \int_{0}^{\infty} t^{m/2} \frac{\sin \sqrt{t}}{\sqrt{t}} \, dt &= \Gamma\left(\frac{m}{2}+1\right)\phi\left(-\frac{m}{2}-1\right) \\ &= \frac{\Gamma\left(\frac{m}{2}+1\right)\Gamma\left(-\frac{m}{2}\right)}{\Gamma\left(-m\right)} = \frac{\Gamma\left(\frac{m}{2}+1\right)\Gamma\left(-\frac{m}{2}\right)}{\Gamma(m+1)\Gamma\left(-m\right)}\Gamma(m+1)\\ &= \frac{\sin \pi m}{\sin \left( \frac{\pi m}{2} \right)} \Gamma(m+1) = 2\cos\left(\frac{\pi m}{2} \right) \Gamma(m+1). \end{align*}
Therefore we have
$$ \int_{0}^{\infty} x^{m} \sin x \, dx =\cos\left(\frac{\pi m}{2} \right) \Gamma(m+1). $$
And here is a rigorous proof: By integration by parts, for $0 < a < b < \infty$,
\begin{align*} \int_{a}^{b} x^{m} \sin x \, dx &= \left[ x^{m} (1 - \cos x) \right]_{a}^{b} - \int_{a}^{b} m x^{m-1} (1 - \cos x) \, dx. \end{align*}
Since $ \Re m \in (-2, 0)$, $x^{m} (1 - \cos x)$ tends to $0$ either as $x \to 0^{-}$ or $x \to \infty$. This proves
$$ \lim_{\substack{a &\to& 0 \\ b &\to& \infty}} \left[ x^{m} (1 - \cos x) \right]_{a}^{b} = 0. $$
Also, by noting that $0 \leq 1 - \cos x \leq 1 \wedge x^2 $, we have
$$ \left| m x^{m-1} (1 - \cos x) \right| \leq |m| x^{\Re m-1} (1 \wedge x^{2}). $$
Since the RHS is integrable, the same is true for the LHS $m x^{m-1} (1 - \cos x)$. Thus the integral
$$ \int_{0}^{\infty} x^{m} \sin x \, dx $$
exists as (at least) in improper sense, and we have
\begin{align*} \int_{0}^{\infty} x^{m} \sin x \, dx &= - \int_{0}^{\infty} m x^{m-1} (1 - \cos x) \, dx \\ &= - \frac{m}{\Gamma(1-m)} \int_{0}^{\infty} \frac{\Gamma(1-m)}{x^{1-m}} (1 - \cos x) \, dx \\ &= - \frac{m}{\Gamma(1-m)} \int_{0}^{\infty} \left( \int_{0}^{\infty} t^{-m}e^{-xt} \, dt \right) (1 - \cos x) \, dx. \end{align*}
But since
\begin{align*} \int_{0}^{\infty} \int_{0}^{\infty} \left| t^{-m}e^{-xt} (1 - \cos x) \right| \, dtdx &\leq \int_{0}^{\infty} \int_{0}^{\infty} t^{-\Re m}e^{-xt} (1 - \cos x) \, dtdx \\ &= \int_{0}^{\infty} \frac{\Gamma(1-\Re m)}{x^{1-\Re m}} (1 - \cos x) \, dtdx < \infty, \end{align*}
Fubini's theorem shows that
\begin{align*} \int_{0}^{\infty} x^{m} \sin x \, dx &= - \frac{m}{\Gamma(1-m)} \int_{0}^{\infty} \int_{0}^{\infty} t^{-m}e^{-xt} (1 - \cos x) \, dxdt \\ &= - \frac{m}{\Gamma(1-m)} \int_{0}^{\infty} \frac{t^{-m-1}}{t^2 + 1} \, dt. \end{align*}
Applying the beta function identity and the Euler reflection formula for the Gamma function, it follows that
\begin{align*} \int_{0}^{\infty} x^{m} \sin x \, dx &= \frac{\pi m}{2\Gamma(1-m)} \csc \left( \frac{\pi m}{2} \right) = \cos \left( \frac{\pi m}{2} \right) \Gamma(m+1) \end{align*}
as desired.