I want to compute the integral \begin{equation} S(t) = \frac{1}{2\pi i}\int_{\epsilon - i\infty}^{\epsilon+i\infty}\frac{e^{st}}{s^{3/2}(1-e^{-2\pi\sqrt{s}})} \; ds \end{equation}
Consider a contour as shown in the figure below.
We can use Cauchy's theorem to get \begin{equation} S_1(t') + S_2(t') + S_3(t') + S_4(t') + S_5(t') + S_6(t') = 0 \end{equation} where $S_1 = S$ is the desired integral, the other terms share the same integrand but use different sections of the contour (with $C_n$ the contour for $S_n$).
One can check that $S_2(t) = S_6(t) = 0$. We examine $S_2$ first. Let $s = Re^{i\phi}, \pi/2<\phi<\pi$ so $\cos\phi<0$ and $\cos(\phi/2)>0$. First we check a limit that will be useful:
\begin{equation} \lim_{R\to\infty}\left|\frac{\sqrt{R} e^{R e^{i\phi}}}{\exp(-2\pi R^{1/2}e^{i\phi/2})}\right| = \lim_{R\to\infty}\left|\frac{\sqrt{R} e^{R \cos\phi}}{\exp(-2\pi R^{1/2}\cos(\phi/2))}\right| = 0. \end{equation} Now we look at a part of the integrand \begin{align*} \lim_{R\to\infty}\frac{e^{Re^{i\phi}}}{1-\exp(-2\pi R^{1/2}e^{i\phi/2})} &= \lim_{R\to\infty}\frac{e^{i \phi}}{(-1)(-2\pi e^{i\phi/2})\frac{1}{2\sqrt{R}}}\frac{e^{R e^{i\phi}}}{\exp(-2\pi R^{1/2}e^{i\phi/2})} \\ &= \lim_{R\to\infty}\frac{e^{i \phi/2}}{\pi}\frac{\sqrt{R} e^{R e^{i\phi}}}{\exp(-2\pi R^{1/2}e^{i\phi/2})} = 0 \end{align*} where we use L'Hopital's rule. The integral for $S_2$ has this term multiplied by a $1/\sqrt{R}$, so it will be zero too.
\begin{equation} \lim_{R\to\infty}\int_{\pi/2}^{\pi} \frac{e^{Re^{i\phi}} iRe^{i\phi}}{(Re^{i\phi})^{3/2}(1-\exp(-2\pi R^{1/2}e^{i\phi/2}))} \;d\phi = 0. \end{equation}
For $S_6$, we will have $s = Re^{i\phi}, \pi<\phi<3\pi/2$ so $\cos\phi<0$ and $\cos(\phi/2)<0$. Since $\cos(\phi/2) < 0$, the denominator in the integrand approaches $\infty$ as $R\to\infty$. Therefore, $S_6(t) = 0$ too.
Now define \begin{equation} 2\pi i\frac{d S_n(t)}{d t} = \int_{C_n}\frac{e^{st}}{s^{1/2}(1-e^{-2\pi\sqrt{s}})} \; ds = I_n(t). \end{equation} Let us examine the semicircular arc $C_4$ parameterised as $s = \varepsilon e^{i\phi}$ with $-\pi/2 \leq \phi \leq \pi/2$. \begin{align} I_4(t) &= \int_{\pi/2}^{-\pi/2}\frac{e^{\varepsilon t e^{i\phi}}i\varepsilon e^{i\phi}}{\sqrt{\varepsilon} e^{i\phi/2}(1-e^{-2\pi\sqrt{\varepsilon}e^{i\phi/2}})} \; d\phi \\ &= \int_{\pi/2}^{-\pi/2}\frac{i\sqrt{\varepsilon}e^{\varepsilon t e^{i\phi}+i\phi/2}}{1-e^{-2\pi\sqrt{\varepsilon}e^{i\phi/2}}} \; d\phi \end{align} The integrand goes to zero almost everywhere in the limit $\varepsilon\to 0$, apart from the point $\phi = 0$. At $\phi = 0$, we will have $\lim_{\varepsilon\to 0} \sqrt{\varepsilon}/(1-e^{-2\pi\sqrt{\varepsilon}}) = 1/2\pi$ which is finite. So the integral $I_4$ is zero, hence $S_4$ is constant. We ignore this value for now and proceed to look at the lines.
Let $s = e^{i\pi} u^2$ for the upper line $C_3$, so $\sqrt{s} = iu$. \begin{align*} I_3(t) &= \int_{\infty}^{0}\frac{e^{-u^2t}}{i u(1-e^{-2\pi iu})} \;(-2u)\; du \\ &= -2i\int_{0}^{\infty}\frac{e^{-u^2t}}{1-e^{-2\pi iu}} \; du. \end{align*} For the lower line $C_5$ let $s = e^{-i\pi} u^2$, so $\sqrt{s} = -iu$. \begin{align*} I_5(t) &= \int_{0}^{\infty}\frac{e^{-u^2t}}{-iu(1-e^{2\pi iu})} \;(-2u)\; du \\ &= -2i\int_{0}^{\infty}\frac{e^{-u^2t}}{1-e^{2\pi iu}} \; du. \end{align*} Adding the two integrals gives \begin{align*} I_3(t) + I_5(t) &= -2i\int_{0}^{\infty}e^{-u^2t}\left[ \frac{1}{1-e^{2\pi i u}} + \frac{1}{1-e^{-2\pi i u}} \right] \; du \\ &= -2i\int_{0}^{\infty}e^{-u^2t}\left[ \frac{-1}{e^{2\pi i u}-1} + \frac{e^{2\pi i u}}{e^{2\pi i u}-1} \right] \; du \\ &= -2i\int_{0}^{\infty}e^{-u^2t} \; du = -i\sqrt{\frac{\pi}{t}} \tag{$t > 0$} \end{align*}
At this point it is clear that something has gone terribly wrong as integrating this result will give $S(t)=A\sqrt{t}+c$, which is clearly not the answer. I don't understand which step(s) are wrong though...
Note: This solution is very much in the vein of the one given by Ron Gordon under Complicated Inverse Laplace Transform (including the figure), since I was unable to apply the result proved there directly to this problem.
There are two mistakes with the original approach:
One can rectify that in the following manner (again, a major part of this solution was pointed out by reuns in the comments).
Let the concatenation $C_3+C_4+C_5 = \gamma$ for convenience.
$$I_+(t) = \int_\gamma \frac{e^{s t}}{s^{1/2}(1-e^{-2\pi\sqrt{s}})} \;ds = \int_{\sqrt{\gamma}}\frac{2e^{u^2t}}{1-e^{-2\pi u}}\;du$$
where $\sqrt{\gamma}$ is homotopic to a vertical line from $\epsilon+i\infty$ to $\epsilon-i\infty$. To close the contour, consider the integral:
$$I_-(t) = \int_{-\sqrt{\gamma}} \frac{2u^2t}{1-e^{-2\pi u}}\;du$$
The sum of the two integrals can be evaluated using the residue theorem. The poles are on the imaginary axis at $ni$, $n\in\mathbb{Z}$ and all have order 1.
\begin{align} I_+(t) + I_-(t) &= -2\pi i\sum_{-\infty}^\infty \lim_{u\to ni}\frac{2(u-ni)e^{u^2 t}}{1-e^{-2\pi u}} \\ &= -4\pi i\sum_{-\infty}^\infty \lim_{u\to 0}\frac{ue^{(u+ni)^2 t}}{1-e^{-2\pi u}} \\ &= -2i\sum_{-\infty}^\infty e^{-n^2 t} \end{align}
where we took an overall minus sign because our loop is clockwise, whereas the residue is defined for counter-clockwise loops.
Evaluate the difference of the two integrals now:
\begin{align} I_+(t) - I_-(t) &= \int_{\sqrt{\gamma}} \frac{2e^{u^2t}}{1-e^{-2\pi u}}\;du - \int_{-\sqrt{\gamma}} \frac{2e^{u^2t}}{1-e^{-2\pi u}}\;du \\ &= \int_{\sqrt{\gamma}} \frac{2e^{u^2t}}{1-e^{-2\pi u}}\;du - \int_{\sqrt{\gamma}} \frac{2e^{u^2t}}{1-e^{2\pi u}}\; (-du) \\ &= \int_{\sqrt{\gamma}} 2e^{u^2t}\left[\frac{1}{1-e^{-2\pi u}} + \frac{1}{1-e^{2\pi u}}\right]\;du \\ &= \int_{\sqrt{\gamma}} 2e^{u^2t}\;du \\ &= \int_{\infty}^{-\infty} 2e^{(\epsilon+iy)^2t}\;i dy \\ &= -2i \sqrt{\pi/t} \end{align}
Combining the two results gives
$$I_+(t) = -i\left[\sum_{-\infty}^\infty e^{-n^2t}+\sqrt{\pi/t}\right]$$
From the first contour integral ($\sum S_i = 0$) and definition of $I_+(t)$, we get that \begin{align} S(t) &= -\frac{1}{2\pi i}\int I_+(t)\;dt + C \\ &= \frac{1}{2\pi}\left[2\sqrt{\pi t}+t-2\sum_{n=1}^{\infty}\frac{e^{-n^2t}}{n^2}\right]+C. \end{align} The only pending task is to figure out the constant $C$; for this we can evaluate the integral for $S(0)$ directly.
\begin{equation} 2\pi i S(0) = \int_{\epsilon - i\infty}^{\epsilon+i\infty}\frac{1}{s^{3/2}(1-e^{-2\pi\sqrt{s}})} \; ds = \int_h\frac{2}{u^2(1-e^{-2\pi u})} \; du \end{equation} where $h$ is a hyperbola arm with asymptotes $z$ and $\bar{z}$. Instead of looking at the hyperbola, we look at line segments parallel to the latus rectum (let $u = x+iy$): \begin{align*} &\lim_{x\to\infty}\int_{-x}^{x}\frac{2i}{(x+iy)^2(1-e^{-2\pi (x+iy)})} \;dy \\ &= \lim_{x\to\infty}\int_{-\pi/4}^{\pi/4}\frac{2ix\sec^2\theta}{x^2(1+i\tan\theta)^2(1-e^{-2\pi x(1+i\tan\theta)})} \;d\theta \\ &= \lim_{x\to\infty}\int_{-\pi/4}^{\pi/4}\frac{2i\sec^2\theta}{x(1+i\tan\theta)^2(1-e^{-2\pi x(1+i\tan\theta)})} \;d\theta = 0 \end{align*} While we haven't been careful with the upper and lower limits, so the endpoints of the integral are not on the hyperbola, we can fix them to be $\pm(\pi/4 - \varepsilon(x))$ with $\varepsilon(x)$ defined appropriately and $\lim_{x\to\infty} \varepsilon(x) = 0$. This will not affect the conclusion. Hence, we get $S(0) = 0$. This gives the result
$$ S(t) = \frac{1}{2\pi}\left[2\sqrt{\pi t}+t+2\sum_{n=1}^{\infty}\frac{1-e^{-n^2t}}{n^2}\right]. $$