Below is a plot of
$$\dfrac{1}{x}\sum_{n=1}^{x}x\ (\mathbb{mod}\ n)-\dfrac{x}{5.6325}$$
where $5.6325$ is very close to whatever the constant actually is. Does anyone know what this constant might be?
Update
Thanks to Hagen von Eitzen's answer & Michael Stocker's comment, it seems that
$$\dfrac{1}{x}\sum_{n=1}^{x}x\ (\mathbb{mod}\ n)-x\left(1-\dfrac{\pi^2}{12}\right)+\dfrac{1}{\pi}$$
stays remarkably tightly bounded. Is there any documentation anywhere about these bounds?
Note: the $+\dfrac{1}{\pi}$ is not stated in Hagen von Eitzen's answer - purely conjectured at this point.
For $n>\frac12 x$, the summand is $x-n$, so these contribute $\approx \sum_{k=0}^{x/2}k\approx \frac18x^2$ to the sum.
For $\frac12x\ge n>\frac13x$, the summand is $x-2n$, so these contribute $\approx\sum_{k=0}^{x/6} 2k$ or $\approx\sum_{k=0}^{x/6} (2k+1)$, at any rate this is $\frac1{36}x^2+O(x)$.
More generally, for large enough $x$ and if $\frac1mx\ge n>\frac1{m+1}x$ with $m\ll x$ (say, $m\le \sqrt x$), the contribution is essentially (i.e. up to $O(x)$) the sum of all multiples of $m$ that are $<\frac x{m+1}$, so $\frac1{2m(m+1)^2}x^2+O(x)$. On the other hand, if we cannot achieve $m\le\sqrt x$, then $n\le \sqrt x$, and those $n$ contribute at most ${\sqrt x\choose 2}\approx \frac 12x$. Thus if we divide the sum by $x^2$, we are left with something like $$\tag1\frac{\sum_{n=0}^xx\bmod n}{x^2} =\sum_{m=1}^{\sqrt x}\frac1{2m(m+1)^2}+O(x^{-1/2}).$$ While this does not recognize your number "by name", it simpifies its numerical approximation: $$ \sum_{m=1}^{\infty}\frac1{2m(m+1)^2}\approx 0.17753296657\ldots \approx \frac1{5.6327566\ldots} $$
Thanks to Michael Stocker's comment we know the exact value: $$1-\frac{\pi^2}{12} $$ (in fact, we somewhat recognize the well-known $\sum \frac1{m^2}=\frac{\pi^2}6$, and ypercube noticed how to arrive at the result via $\frac1{m(m+1)^2}=\underbrace{\frac1m-\frac1{m+1}}_{\text{telescope}}-\frac1{(m+1)^2}$). Now all that remains is to get a better grip on the Big-Oh' I allowed to enter into the calculation ... Indeed, the OP's observation suggests that $$\tag2\frac{\sum_{n=0}^xx\bmod n}{x^2} =1-\frac{\pi^2}{12}+O(x^{-1})$$ and the worse estimate in $(1)$ cannot come from the incomplete series alone.