I've written the Fourier sine transform $\mathcal{F}_s(f)(\nu)$ of a function $f(x)$ (see its definition from this Wikipedia) using the Proposition 1 of [1], the Prime Number Theorem, the change of variable $x=\nu t$ and integration.
For a large integer $N>1$, I define the approximants $$\mathcal{A}_N(f)(\nu):=-\pi\int_{-\infty}^\infty \left(\sum_{n=1}^N \frac{\mu(n)}{n} \left\{ n\nu t \right\} \right)f(t)dt\tag{1}$$ where thus $\mu(n)$ is the Möbius function, and $ \left\{ x \right\} $ is the fractional part function; and being $$g(\nu):=\mathcal{F}_s(f)(\nu)$$ the Fourier sine transform of the function $f(x)$.
You can read the definition of the Möbius function from some free available online encyclopedia.
Question. In this situation I would like to know what kind of convergence can I presume for the limit $$\mathcal{A}_N(f)(\nu)\to g(\nu)\tag{2}$$ as $N\to\infty$. Alternatively/additionally quantify $$|\mathcal{A}_N(f)(\nu)-g(\nu)|\tag{3}$$ as $N\to\infty$. Many thanks.
References:
You can read next reference from $\mathcal{The}$ EUROPEAN DIGITAL MATHEMATICS LIBRARY.
[1] S. Segal, On an identity between infinite series of arithmetic functions, Acta Arithmetica (1976), Volume: 28, Issue: 4, pp. 345-348.
The OP meant that from the Fourier series $$S(x) = -\pi (x-\lfloor x \rfloor) = -\frac{\pi}{2}+\sum_{n=1}^\infty \frac{\sin(2 \pi n x)}{ n}$$ And $\sum_{n=1}^\infty \frac{\mu(n)}{n} = 0$ (equivalent to the PNT)
We get $$\sum_{n=1}^\infty \frac{\mu(n)}{n} S(nx) = \sin(2 \pi x)$$ The convergence being at least in $L^2([0,1])$.
Thus for any nice enough function $f$ $$\int_0^\infty f(x) \sin(2 \pi x)dx = \lim_{N \to \infty} \int_0^\infty (\sum_{n=1}^N \frac{\mu(n)}{n} S(nx)) f(x)dx \tag{1}$$ Indeed, the set for which $(1)$ is true encodes the Riemann hypothesis.
Taking $f(x) = x^{-\sigma-1}$, as $\int_0^\infty S(x) x^{-s-1}dx = \frac{\pi}{s} \zeta(s)$ for $\Re(s) \in (0,1)$ we see that $$\int_0^\infty (\sum_{n=1}^N \frac{\mu(n)}{n} S(nx)) x^{-\sigma-1}dx = \sum_{n=1}^N \frac{\mu(n)}{n} \int_0^\infty S(nx) x^{-\sigma-1}dx$$ $$ = \sum_{n=1}^N \frac{\mu(n)}{n} \int_0^\infty S(y) (y/n)^{-\sigma-1}d(y/n)= \frac{\pi}{s} \zeta(s) \sum_{n=1}^N \mu(n) n^{\sigma-1}$$ Thus $\int_0^\infty x^{-s-1} \sin(2 \pi x)dx = \lim_{N \to \infty} \int_0^\infty (\sum_{n=1}^N \frac{\mu(n)}{n} S(nx)) x^{-\sigma-1}dx $ iff $\sum_{n=1}^\infty \mu(n) n^{\sigma-1}$ converges (to $\frac{1}{\zeta(1-\sigma)}$) iff $\zeta(s)$ has no zeros for $\Re(s)\ge 1-\sigma$.
This is often mentioned as the functional analysis criteria for the Riemann hypothesis.