First, let us use $t=\left[\,t\right]+\left\langle\,t\right\rangle $, for integral and fractional part of $t$ respectively. I try to prove that $$\int_N^\infty \frac{\left\langle\,t\right\rangle -\frac{1}{2}}{t}dt=\mathcal O(1/N).$$
I guess it is true because I need this to prove Stirling formula, $$\exp\left(-\int_N^\infty\frac{\left\langle\,t\right\rangle -\frac{1}{2}}{t}dt\right)=1+\mathcal O\left(\frac{1}{N}\right)$$ Here are what I attempted. I tried with the range $[N,N+1)$.
$$\int_N^{N+1}\frac{\left\langle\,t\right\rangle -\frac{1}{2}}{t}dt=\int_N^{N+1}\left\{1-\frac{1}{2t}\right\}dt-\int_{N}^{N+1}\frac{\left[\,t\right] }{t}dt=1-\frac{1}{2}\log\left(1+\frac{1}{N}\right)-A$$
The first one is quite straight forward but the second I am not sure if I can do this (I cover the integral only for the number close to $N+1$ frome the left then sum up with this similar cases to infinity.)
$$A=N\int_N^{(N+1)^-}\frac{dt}{t}=N\log\left(1+\frac{1}{N}\right)$$
Then with the $\log$ expansion I get the integral of this range (if the above valids), $$1-\frac{1}{2}\left(\frac{1}{N}-\frac{1}{2N^2}+\frac{1}{3N^3}-\cdots\right)-N\left(\frac{1}{N}-\frac{1}{2N^2}+\frac{1}{3N^3}-\cdots\right)=\mathcal O\left(\frac{1}{N^2}\right)$$
Thus, on sum up all ranges with some formula in the last inequality (this establishes the first inequality above I suppose) $$\int_N^\infty \frac{\left\langle\,t\right\rangle -\frac{1}{2}}{t}dt=\mathcal O\left(\sum_{n\ge N} \frac{1}{n^2}\right)=\mathcal O\left(\frac{1}{N}\right)$$
Assuming the integral understood in the improper sense (that is, $\int_N^\infty=\lim\limits_{T\to\infty}\int_N^T$) as we have to, your approach is perfectly valid; the last step may be justified by $\sum_{n>N}\frac1{n^2}<\sum_{n>N}\frac1{n(n-1)}=\frac1N$.
Another idea is to use integration by parts, with the fact that $\phi(x)=\int_0^x\left(\langle t\rangle-\frac12\right)dt$ is periodic (with a period $1$), hence bounded (in fact $-1/8\leqslant\phi(x)\leqslant 0$ for all $x$). We get $$\int_N^\infty\left(\langle t\rangle-\frac12\right)\frac{dt}{t}=\left.\frac{\phi(t)}{t}\right|_N^\infty+\int_N^\infty\frac{\phi(t)}{t^2}\,dt=\int_N^\infty\frac{\phi(t)}{t^2}\,dt,$$ and the absolute value of this integral is at most $\frac18\int_N^\infty\frac{dt}{t^2}=\frac1{8N}$.