There is a hint which says $\left|\int\limits_0^{\infty} \cos\left\{{a\over2}\left(x+{1\over x}\right)\right\}x^{k-2}\ dx\right|\le C a^{k-2}$ where C is some constant.
I somehow feel that I need to use the Gamma function to show the convergence of this integral, but I'm not getting any fruitful approx to solve the problem. I have tried to apply different tests. Can anyone give a idea or hint to show the convergence? Thanks for help in advance.
The (improper) integral converges if and only if $0<k<2$.
Indeed, considering $a>0$ fixed, for $\lambda\in\mathbb{R}$ and $x\geqslant 1$, put \begin{align*} C(x)&:=\cos\frac{a}2\left(x+\frac1x\right), &C_\lambda(x)&:=\int_1^x t^\lambda C(t)\,dt, \\S(x)&:=\sin\frac{a}2\left(x+\frac1x\right), &S_\lambda(x)&:=\int_1^x t^\lambda S(t)\,dt, \end{align*} so that the given integral equals $C_{k-2}(+\infty)+C_{-k}(+\infty)$. It remains to show that $C_\lambda(+\infty)$, as well as $S_\lambda(+\infty)$, exist iff $\lambda<0$. This is clear for $\lambda<-1$ (absolute convergence); further, \begin{align*} \frac{a}2\big(C_\lambda(x)-C_{\lambda-2}(x)\big)&=x^\lambda S(x)-S(1)-\lambda S_{\lambda-1}(x), \\\frac{a}2\big(S_{\lambda-2}(x)-S_\lambda(x)\big)&=x^\lambda C(x)-C(1)-\lambda C_{\lambda-1}(x), \end{align*} using integration by parts. This implies the existence for $\lambda<0$, and for $\lambda\geqslant 0$ we prove \begin{align*} C_\lambda(x)&=(2/a)x^\lambda S(x)+\text{const.}+o(x^\lambda),\\ S_\lambda(x)&=-(2/a)x^\lambda C(x)+\text{const.}+o(x^\lambda) \end{align*} as $x\to+\infty$ (using, say, induction on $\lfloor\lambda\rfloor$ and the above).