This is the Riemann function.
I would like to determine its asymptotics as $t \rightarrow 0^+ $.
First let $t=x^2 $, so that we treat the series
$$ \sum_{n\geq 1 } \frac{\sin n^2 x^2}{ n^2}. $$
We use the Mellin transform. For that, we need the Mellin transform the function $\sin x^2$. By change of variables, and using the fact that the Mellin transform of the function $\sin x $ is $\Gamma(s) \sin \frac{\pi s }{2}$, we get
$$ \mathcal{M}[\sin x^2 ;s ] = \frac{1}{2}\Gamma(\frac{s}{2}) \sin\frac{\pi s }{4} , \;\;\;0 < Re(s) < 2. $$
The Mellin transform the original series is then
$$ \frac{1}{2}\Gamma(\frac{s}{2}) \sin\frac{\pi s }{4} \zeta(s+2) , \;\;\;0 < Re(s) < 2. $$
Now we use the inversion formula to recover the original series. It is
$$\frac{1}{2\pi i }\int_C x^{-s} \frac{1}{2}\Gamma(\frac{s}{2}) \sin\frac{\pi s }{4} \zeta(s+2) ds , $$
where we can take the integration contour $C$ to be the vertical line from $1-i\infty $ to $1+ i\infty $.
The common trick is then to shift the integration line to left. In this process, the poles of the integrand are rounded up one by one.
Now, the problem is that the gamma function has poles at $s = 0, -2, -4 ,-6,$ etc. The zeta function has a single pole at $s = -1$. The sine function has zeros at $s = 0 , -4 , -8, $ etc. The zeta function has zeros at $s = -4, -6, -8,$ etc.
After the cancellation, the rest poles are $s = -1 , -2 $. We get a quadratic polynomial of $x$.
This is obviously wrong. The Riemann function is non-differentiable almost everywhere.
Could anyone show me where I made a mistake?
You didn't check the growth of $\frac{1}{2}\Gamma(\frac{s}{2}) \sin\frac{\pi s }{4} \zeta(s+2)$ on vertical lines which is needed to shift the contour on the left and deduce an asymptotic.
You can look at $$f_k(u)=f \ast \frac{k}{\sqrt{2\pi}}e^{- k^2 u^2/2}$$ where $$f(u) = \sum_{n=1}^\infty \frac{\sin(n^2 e^{-u})}{n^2}$$ then $$\mathcal{L}[f_k(u)](s) = \sin(\pi s/2)\Gamma(s) e^{s^2/(2k^2)} \zeta(2s+2)$$ this time everything is $L^1$ on vertical lines so that as $u \to \infty$, $$f_k(u) =\frac{1}{2i\pi} \int_{(-1/2-\epsilon)}\mathcal{L}[f_k(u)](s) e^{su}ds+ Res(\mathcal{L}[f_k(u)](s),s=-1/2)\\= \frac{\sqrt{2\pi}}{2} e^{1 /(8k^2)} e^{-u/2}+O(e^{-(1/2+\epsilon)u})$$
Since $f$ is uniformly continuous on $[a,\infty)$ and bounded then $f_k \to f$ uniformly on $[a,\infty)$. Then we can check if the same holds for $f_k u^{-m} e^{ub}\to f u^{-m}e^{ub} $ for $b \le 1/2$