Let $f(x)$ be convex over $[1,+\infty)$ and decrease monotonically to zero, with $f(1)=1$ and $f\left(\frac{3}{2}\right)=\frac{2}{3}.$ Prove $$-\frac{1}{8}<\int_1^{+\infty} \left(x-\lfloor x \rfloor -\frac{1}{2}\right)f(x)\,{\rm d}x<-\frac{1}{18}.$$
Indeed, $\varphi(x):=x-\lfloor x \rfloor -\frac{1}{2}$ is a periodic function with period $1$. Over $[1,2)$, $$\begin{align}\varphi(x)=x-1/2\,(1\leq x < 2),\end{align}$$ and in general $$\begin{align}\varphi(x)=x-n-1/2\,&( n\leq x < n+1)\end{align}$$ But how to estimate the integral in general?
This is a partial answer: I can get a better lower bound but can't quite get the upper bound. I'm wondering if OP can provide more context for this question, as it's old and somewhat difficult. Nevertheless, here are my thought so far.
Note that the integral converges by Dirichlet's Test. The idea is to use the convexity of $f$ to analyze the integral by looking at piecewise-linear parts.
Since we are guaranteed convergence, we can split the integral into unit-length intervals. For clarity I will write $\infty$ as the upper limit of summation, but really one must take a limit of partial sums, viz $\lim_{M\to\infty}\sum_{k}^M$, as $f(x)=1/x$ shows. Nevertheless, we have: $$I=\int _{1}^{\infty}\varphi(x) f(x)\,dx = \sum_{k=1}^{\infty}\int _{k}^{k+1}\varphi(x) f(x)\,dx$$
Use the periodicity of $\varphi$ and make the substitution $x-(k-1)=t$:
$$= \sum_{k=1}^{\infty}\left(\int _{1}^{2}\varphi(t) f(t+k-1)\,dt\right)$$ $$= \sum_{k=0}^{\infty}\left(\int _{1}^{2}\varphi(t) f(t+k)\,dt\right)$$
Put $\varphi(t)=t-3/2$ and split the range of integration. Importantly, $\varphi<0$ on $(1,3/2)$ and $\varphi > 0$ on $(3/2,2)$, while $f$ is always non-negative. $$=\sum_{k=0}^{\infty}\left(\int _1^{3/2}(t-3/2) f(t+k)\,dt+\int _{3/2}^{2}(t-3/2) f(t+k)\,dt\right)$$
Suppose we want to minimize $I$. Then we want to maximize the negative contribution over $(1,3/2)$ and minimize the positive contribution over $(3/2,2)$. Since we know $f$ is convex, on the first part it suffices to assume $f$ is linear. On the second part, an initial lower bound is $f(k+2)$. However, when we move to the next index, this positive contribution will outweigh the next negative contribution. So, in fact we can simplify things by setting $f(x)\equiv 0$ for $x>3/2$. Then our estimate becomes: $$ I >\int _1^{3/2}(t-3/2)((3-2t)f(1)+2(t-1)f(3/2))\,dt $$ This looks a mess, but it's actually easy to integrate: $$ I> -\frac{1}{12}f(1)-\frac{1}{24}f(3/2) =\frac{-1}{9} $$For an upper bound, again we examine $k=0$ and discard further terms of the sum. Here we want to minimize the negative contribution, so on $(1,3/2)$ we will use $f(x)>f(3/2)=2/3$. This will serve as an upper bound on $(3/2,2)$ as well, giving $$ I < \int _1^{3/2}(t-3/2)((3-2t)f(1)+2(t-1)f(3/2))\,dt+\int _{3/2}^2(t-3/2)f(3/2)\,dt $$A quick integration gives $$I < \frac{-1}{36};$$ however, OP claims that $-1/18$ is possible. Perhaps my argument can be refined or perhaps different techniques are needed.