In looking for an efficient way to show the relation
$$\int_{-\infty}^\infty \frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \, dx = \frac{(2 \cos n/2)^{\alpha+\beta-2}}{\Gamma(\alpha+\beta-1)} e^{ \frac{in(\beta-\alpha)}{2}} \text{ or }~~ 0$$
acc. as $|n|< \pi$ or $|n|>\pi,$ respectively, and $\operatorname{Re}(\alpha+\beta)> 1,$ it occurred to me that this looks to be the Fourier transform (or inverse transform ) of a sinc function, and then I noticed in a Wiki article that
$$\frac{\sin(\pi x)}{ \pi x} = \frac{1}{\Gamma(1+x)\Gamma(1-x)}.$$
I am familiar with the basic properties like linearity and scaling for transform pairs and have found some obvious points but seem far from finding a simple way to modify the normalized sinc function above to give the desired rectangular function.
Can someone tell me what the correct form of the sinc function should be?
Thanks for any help.
Well, I have a little different result, but still hope this will help. I think the application of beta-function will be fruitful enough.
Just remember, that $$\mathrm{B}(\alpha+x,\beta-x)=\frac{\Gamma(\alpha+x)\Gamma(\beta-x)}{\Gamma(\alpha+\beta)}$$ $$ \frac{1}{\Gamma(\alpha+x)\Gamma(\beta-x)}=\frac{1}{\Gamma(\alpha+\beta)}\frac{1}{\mathrm{B}(\alpha+x,\beta-x)}$$ So: $$\int_{-\infty}^\infty \frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \, dx=\frac{1}{\Gamma(\alpha+\beta)}\int_{-\infty}^\infty \frac{e^{inx}}{\mathrm{B}(\alpha+x,\beta-x)} \, dx$$ Then one can use the definition of the beta-function (5.12.6): $$\int _{0}^{\pi}(\mathop{\sin}\nolimits t)^{{a-1}}e^{{ibt}}dt=\frac{\pi}{2^{{a-1}}}\frac{e^{{i\pi b/2}}}{a\mathop{\mathrm{B}}\nolimits\!\left(\frac{1}{2}(a+b+1),\frac{1}{2}(a-b+1)\right)}$$ Setting $a=\alpha+\beta-1, \ b=\alpha-\beta+2x$, after simplifications one will obtain: $$\frac{1}{\mathrm{B}(\alpha+x,\beta-x)}=\frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{\pi}(\alpha+\beta-1)\int _{0}^{\pi}(\mathop{\sin}\nolimits t)^{{\alpha+\beta-2}}e^{{i(\alpha-\beta+2x)t}}dt,$$ which is true, when $\mathrm{Re}(a)>0$, or $\mathrm{Re}(\alpha+\beta)>1$.
So the initial integral will look like (here you already can see the terms as in the answer ;)): $$\int_{-\infty}^\infty \!\frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \!\,\! dx= \frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{\pi}\frac{(\alpha+\beta-1)}{\Gamma(\alpha+\beta)} \int_{-\infty}^\infty \!e^{inx} \!\!\int _{0}^{\pi}(\mathop{\sin}\nolimits t)^{{\alpha+\beta-2}}\!e^{{i(\alpha-\beta+2x)t}}\!dt \ dx $$
$$\int_{-\infty}^\infty \!\frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \!\,\! dx= \frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{\pi \ \Gamma(\alpha+\beta-1)} \int _{0}^{\pi} \!(\mathop{\sin}\nolimits t)^{{\alpha+\beta-2}} e^{{i(\alpha-\beta)t}} \int_{-\infty}^\infty \!e^{i(n+2t)x} \!dx \ dt $$ The inside integral is the definition of the Dirac delta function: $$\int_{-\infty}^\infty \!e^{i(n+2t)x} \!dx= 2\pi \delta(2t+n)=\pi \delta\bigg(t+\frac{n}{2}\bigg)$$ And if $0\leq -\frac{n}{2}\leq \pi$: $$\int _{0}^{\pi} \!(\mathop{\sin}\nolimits t)^{{\alpha+\beta-2}} e^{{i(\alpha-\beta)t}} \delta\bigg(t+\frac{n}{2}\bigg) dt=\theta (-n) e^{-\frac{1}{2} i n (\alpha -\beta )} \left(-\sin\bigg(\frac{n}{2}\bigg)\right)^{\alpha +\beta -2}$$ where is $\theta (-n)$ the Heaviside theta function.
So finally the result is: $$\int_{-\infty}^\infty \frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \, dx= \frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{ \Gamma(\alpha+\beta-1)}\theta (-n) e^{-\frac{1}{2} i n (\alpha -\beta )} \left(-\sin\bigg(\frac{n}{2}\bigg)\right)^{\alpha +\beta -2} $$ Or after some simplification: $$\int_{-\infty}^\infty \frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \, dx= \frac{(2 \sin(\frac{n}{2}))^{\alpha+\beta-2}}{ \Gamma(\alpha+\beta-1)}e^{i\frac{n}{2}(\beta-\alpha)}\theta (-n) $$
OR
One can use the definition of the beta-function (5.12.5): $$\int _{0}^{\frac{\pi}{2}}(\mathop{\cos}\nolimits t)^{a-1}\cos(bt)dt=\frac{\pi}{2^{{a}}}\frac{1}{a\mathop{\mathrm{B}}\nolimits\!\left(\frac{1}{2}(a+b+1),\frac{1}{2}(a-b+1)\right)}$$ using the same substitutions for $a$ and $b$ and expanding $cos(bt)$ in terms of complex exponents one will get:
$$\int_{-\infty}^\infty \!\!\frac{e^{inx}}{\Gamma(\!\alpha+x)\!\Gamma(\beta-x)} \!\,\! dx= \frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{\pi}\frac{(\alpha+\beta-1)}{\Gamma(\alpha+\beta)} \int_{-\infty}^\infty \!e^{inx} \!\!\int _{0}^{\frac{\pi}{2}}(\mathop{\cos}\nolimits t)^{{\alpha+\beta-2}}\!e^{{i(\alpha-\beta+2x)t}}\!dt \ dx \! + \\ +\frac{2^{\alpha+\beta-2}e^{i\frac{\pi}{2}(\beta-\alpha)}}{\pi}\frac{(\alpha+\beta-1)}{\Gamma(\alpha+\beta)} \int_{-\infty}^\infty \!e^{inx} \!\!\int _{0}^{\frac{\pi}{2}}(\mathop{\cos}\nolimits t)^{{\alpha+\beta-2}}\!e^{{-i(\alpha-\beta+2x)t}}\!dt \ dx $$ Using precisely the same reasoning one will come up with the answer: $$\int_{-\infty}^\infty \frac{e^{inx}}{\Gamma(\alpha+x)\Gamma(\beta-x)} \, dx= \frac{(2 \cos(\frac{n}{2}))^{\alpha+\beta-2}}{ \Gamma(\alpha+\beta-1)}e^{i\frac{n}{2}(\beta-\alpha)}(\theta (n,\pi -n)+\theta (-n,n+\pi )) $$ and $$ \theta (-n,n+\pi )+\theta (n,\pi -n)= \begin{cases} 1, & -\pi <n<\pi \\ 0, & \mathrm{otherwise} \end{cases} $$