I am dealing with the term $\sum_{n \le x} \frac{\mu(n)}{n^2}$ (where $x \to \infty$). Here, $\mu$ denotes the Möbius function. Writing $D_\mu (s) = \sum_{n \in \mathbb{N}} \frac{\mu(n)}{n^s}$ for the associated Dirichlet series, I can rewrite this as
$ \sum_{n \le x} \frac{\mu(n)}{n^2} = D_\mu(2) - \sum_{n > x} \frac{\mu(n)}{n^2} = \frac{6}{\pi^2} - \sum_{n > x} \frac{\mu(n)}{n^2}. $
Now I wonder, how small is the last sum? I am aware that
$ \lvert \sum_{n > x} \frac{\mu(n)}{n^2} \rvert \ll \sum_{n > x} \frac{1}{n^2} \ll \frac{1}{x}, $
but can I do better? If not, is it possible to show $\lvert \sum_{n > x} \frac{\mu(n)}{n^2} \rvert \gg \frac{1}{x}$?
Some comments have already pointed out that the decay of $\sum_{x \le n} \frac{\mu(n)}{n^2}$ is related to the growth of the "Mertens function"
$$M(x) = \sum_{n \le x} \mu(n).$$
I have taken the time now to make this precise. In particular, we will see in the end why it is probably not easy to prove/disprove
$$\lvert \sum_{N < n} \frac{\mu(n)}{n^2} \rvert \gg \frac{1}{N}$$
(which I had hoped when I asked the question).
Let us denote (for real numbers $y \le x$)
$$M(y, x) = \sum_{y \le n \le x} \mu(n),$$
so that $M(x) = M(1, x)$. Then, for integers $N < N'$, a partial summation yields
$$\sum_{N < n \le N'} \frac{\mu(n)}{n^2} = M(N+1, N') \cdot \frac{1}{N'^2} + 2 \int_{N+1}^{N'} M(N+1, t) \cdot \frac{1}{t^3} \text{d} t. $$
Since we have $\lvert M(N+1, N') \rvert \le N' - N \le N'$, the first summand goes to $0$ as $N' \to \infty$. Therefore, taking limits on both sides gives (for $N \in \mathbb{N}$ still fixed) the formula
$$ \sum_{N < n} \frac{\mu(n)}{n^2} = 2 \int_N^\infty M(N+1, t) \cdot \frac{1}{t^3} \text{d} t \\ = 2 \int_N^\infty M(t) \cdot \frac{1}{t^3} \text{d} t - 2 \int_N^\infty M(N) \cdot \frac{1}{t^3} \text{d} t \\ = 2 \int_N^\infty M(t) \cdot \frac{1}{t^3} \text{d} t + M(N) \cdot \frac{1}{N^2}. $$
So if we have $M(N) \ll N^\alpha$ for some $\alpha < 1$, then the right summand and the integral both evaluate to $\ll N^{\alpha - 2}$.
It is clear that $M(N) \ll N$ from the definition of the Mertens function. So this gives an alternative proof for $\sum_{N < n} \frac{\mu(n)}{n^2} \ll \frac{1}{N}$. And if we knew that the Riemannian $\zeta$-function has no zeros in $\Re(s) > \alpha$ for an $\alpha < 1$, then we could show (with a long computation, and using Perron's formula) $M(N) \ll N^\alpha$.
This also proves Conrad's remark in the comments above: If $\lvert \sum_{N < n} \frac{\mu(n)}{n^2} \rvert \gg \frac{1}{N}$, then no such $\alpha < 1$ exists, i.e. the Riemann hypothesis would be false.