Can anybody please guide me in evaluating this expression, for my research work. I have tried a lot but in vain. Both the integrals involve gamma functions and even wolfram said that the time limit exceeded. There might be some technique or rule applicable or some workaround. I am not a mathematics student and thus i am having quite a hard time. Thanks in advance !!
I assume you mean $[h(t)]^2$ rather than $h(h(t))$, when you write $h^2(t)$. Also assuming $f_m$ is constant...
Let $I(b) = \displaystyle \int_0^{\infty} h^2(t) e^{bt} \, dt$. Then $I(-j 2\pi f_m)$ is the numerator and $I(0)$ is the denominator.
$\begin{align} I(b) & = \displaystyle \int_0^{\infty} h^2(t) e^{bt} \, dt\\ & = \displaystyle \int_0^{\infty} t^{4/k} e^{-2at} e^{bt} \, dt\\ & = \displaystyle \int_0^{\infty} t^{4/k} e^{-(2a-b)t} \, dt \end{align}$
Let $(2a - b)t = x$. Then $dt = dx/(2a-b)$.
$\begin{align} I(b) & = \displaystyle \int_0^{\infty} \left(\dfrac{x}{2a - b}\right)^{4/k} e^{-x} \dfrac{1}{2a - b} \, dt\\ & = \dfrac{1}{(2a - b)^{1 + 4/k}}\displaystyle \int_0^{\infty} x^{4/k} e^{-x} \, dt\\ & = \dfrac{1}{(2a - b)^{1 + 4/k}} \Gamma\left(\dfrac{4}{k} + 1\right) \end{align}$
Thus, $\boxed{\dfrac{I(-j 2\pi f_m)}{I(0)} = \left(1 + j\dfrac{\pi f_m}{a} \right)^{-(1 + \frac{4}{k})}}$