I want to study the following fact by Tong which I don't want the exact form like this. I just want something like $\gg X^{3/2}$ or $\ll X^{3/2}$ for the left-hand side. Now, I don't want to read the paper by him but try myself.
It is shown that, for $R(y)=\sum_{n\le y}\tau(n)-y(\log y+2\gamma-1)+O(1)$,
$$\int_0^X R(y)^2dy=\dfrac{\zeta(3/2)^4}{6\pi^2\zeta(3)}X^{3/2}+o(X^{3/2}).$$
Here are what I tried and failed, but I want to know where since I approximate few times (and even equality does not seem true too). At the first moment I just want to grasp an approximation to the problem using Van der Corput's method and ideas in the book. Thank you in advance for any help and clarification. For example, where I shouldn't be using the approximation (as it is not valid or it may have a hidden large error and holes)
Here we look at the book (link), and denote $\lVert x\rVert$ be the nearest distant to integer, so $\lVert 1.2 \rVert=0.2$, $\lVert 0.7\rVert=0.3$. Also, we let $\left\langle\, x\right\rangle$ be the fractional part of $x$, and $e(\alpha)=\exp(2\pi i\alpha)$. Let $B_1(t)=\left\langle\, t\right\rangle -\frac{1}{2}$ and $B(t)=\frac{1}{2J}\int_{-1/J}^{1/J} B_1(t+u) du.$
$$\int_0^X R(y)^2dy=\int_0^X \left(\sum_{q\le \sqrt{y}}B_1\left(\dfrac{y}{q}\right)\right)^2 dy\approx \int_0^X \left(\sum_{q\le \sqrt{y}}B\left(\dfrac{y}{q}\right)\right)^2 dy.$$
$$\therefore \int_0^X R(y)^2dy\approx \sum_{y\le X}\left(\sum_{q\le \sqrt{y}}B\left(\dfrac{y}{q}\right)\right)^2=\sum_{q,q_1\le \sqrt{X}}\sum_{\max(q^2,q_1^2)\le y\le X}B\left(\dfrac{y}{q}\right)B\left(\dfrac{y}{q_1}\right).$$
Thus, we have an ineqaulity which is taken by letting $q\ge q_1$ (and $q<q_1$ will be almost the same)
$$\sum_{q,q_1\le \sqrt{X}}\sum_{\max(q^2,q_1^2)\le y\le X}B\left(\dfrac{y}{q}\right)B\left(\dfrac{y}{q_1}\right)\ll\left|\sum_{q\le \sqrt{X}}\left\{\sum_{q^2\le y\le X}B\left(\dfrac{y}{q}\right)\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right\}\right|$$
By the proof used in Voronoi's theorem $$\left|\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right|\ll X^{1/3}\log X.$$
Edited: This should be $$\left|\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right|\ll\sqrt{X}$$ and the last inequality below is $\ll X^{3/2}\log X$ not contradicting anymore but not strong enough.
We also have, by mimicking the proof, but use instead Theorem 6.7, which states that if $I$ be a bounded interval in $\mathbb R$, and $f\in\mathcal C^1(I,\mathbb R)$ be a real function such that $f'$ is monotonic on $I$ and satisfies $\min_{x\in I}\lVert f'(x)\rVert\ge \lambda>0.$
$$\left|\sum_{n\in I}e\Big(f(n)\Big)\right|\le\dfrac{2}{\pi\lambda},$$
to obtain (by setting the absolute convergent Fourier series of $B(t)$) $$\left|\sum_{y\le X} B\left(\dfrac{y}{q}\right)\right|=\left|\sum_{y\le X} \sum_{j\ge 1}a_j\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\approx\left|\sum_{r\le \frac{\log X}{\log 2}}\sum_{j\ge 1}a_j\sum_{2^r<y \le 2^{r+1}}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|:=R,$$
where $a_j=-\dfrac{J}{2\pi^2j^2}\sin\left(\dfrac{2\pi j}{J}\right)$, and as in the book we have $|a_j|\ll\min(j,J)/j^2$.
$$R\ll \sum_{r\le \log X}\sum_{j\ge 1} |a_j|\left|\sum_{2^r<y\le 2^{r+1}}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{r\le \log X}\sum_{j\ge 1} |a_j|\left|\sum_{2^r<y\le 2^{r+1}}e\left( j\cdot\dfrac{y}{q}\right)\right|.$$
Thus, by theorem 6.7 above and $f(n)=jn/q$ with $I=(2^r,2^{r+1}]$ so $f'(t)=j/q$ be a constant, therby giving $\lambda = \lVert j/q\rVert$,
$$\sum_{j\ge 1}|a_j|\left|\sum_{2^r<y\le2^{r+1}}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{j\le J}'\dfrac{1}{j\lVert j/q\rVert}+\sum_{j>J}' J\cdot\dfrac{1}{j^2\lVert j/q\rVert},$$
where the $\sum'$ denotes the sum over which $q\nmid j.$
Therefore,
$$\sum_{j\ge 1}|a_j|\left|\sum_{2^r<y\le2^{r+1}}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{j\le J}\sum_{k\le q-1}\dfrac{1}{j(k/q)}+\sum_{j>J}\sum_{k\le q-1} J\cdot\dfrac{1}{j^2(k/q)},$$
which is $\ll q\log J\log q+Jq\log q$ and we can choose $J>1$ to be $O(1)$. Now, we plug in and obtain
$$\left|\sum_{q^2\le y\le X} B\left(\dfrac{y}{q}\right)\right|\ll q\log q\log q^2+q\log q\log X\ll q\log q\log X.$$
$$\sum_{q,q_1\le \sqrt{X}}\sum_{\max(q^2,q_1^2)\le y\le X}B\left(\dfrac{y}{q}\right)B\left(\dfrac{y}{q_1}\right)\ll \sum_{q\le \sqrt{X}}\left|\sum_{q^2\le y\le X}B\left(\dfrac{y}{q}\right)\right| \left|\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right|$$
$$\ll \sum_{q\le \sqrt{X}} q\log q\log X\cdot X^{1/3}\log X\ll X^{4/3}\log^3 X=o(X^{3/2}),$$
which is impossible since we will have $\int_0^X R(y)^2 dy\approx o(X^{3/2})$, but contradict the above fact.
It seems to me that using the same setting is possible, but I need to change some of them.
In the Voronoi's section I mentioned, it should be $$\left|\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right|\ll\sqrt{X}.$$
We also have, by mimicking the proof, but use instead Theorem 6.7, which states that if $I$ be a bounded interval in $\mathbb R$, and $f\in\mathcal C^1(I,\mathbb R)$ be a real function such that $f'$ is monotonic on $I$ and satisfies $\min_{x\in I}\lVert f'(x)\rVert\ge \lambda>0,$
$$\left|\sum_{n\in I}e\Big(f(n)\Big)\right|\le\dfrac{2}{\pi\lambda},$$
to obtain (by setting the absolute convergent Fourier series of $B(t)$, and I need no seperation as $r$) $$\left|\sum_{y\le X} B\left(\dfrac{y}{q}\right)\right|=\left|\sum_{y\le X} \sum_{j\ge 1}a_j\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|=\left|\sum_{j\ge 1}a_j\sum_{y\le X}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|:=R,$$
where $a_j=-\dfrac{J}{2\pi^2j^2}\sin\left(\dfrac{2\pi j}{J}\right)$, and as in the book we have $|a_j|\ll\min(j,J)/j^2$.
$$R\ll \sum_{j\ge 1} |a_j|\left|\sum_{y\le X}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{j\ge 1} |a_j|\left|\sum_{y\le X}e\left( j\cdot\dfrac{y}{q}\right)\right|.$$
Thus, by theorem 6.7 above and $f(n)=jn/q$ with $I=(0,X]$ (as $f'(t)$ is a constant in the range) so $f'(t)=j/q$ be a constant, therby giving $\lambda = \lVert j/q\rVert$,
$$\sum_{j\ge 1}|a_j|\left|\sum_{0<y\le X}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{j\le J}'\dfrac{1}{j\lVert j/q\rVert}+\sum_{j>J}' J\cdot\dfrac{1}{j^2\lVert j/q\rVert},$$
where the $\sum'$ denotes the sum over which $q\nmid j.$
Therefore,
$$\sum_{j\ge 1}|a_j|\left|\sum_{y\le X}\sin\left(2\pi j\cdot\dfrac{y}{q}\right)\right|\ll\sum_{j\le J}\dfrac{1}{j(1/q)}+\sum_{j>J} J\cdot\dfrac{1}{j^2(1/q)},$$
which is $\ll q\log J+Jq$ and we can now choose $J>1$ to be $O(1)$ so that it is $\ll q$. Then, we plug in and obtain that
$$\left|\sum_{q^2\le y\le X} B\left(\dfrac{y}{q}\right)\right|\ll q.$$ Furthermore, $$\sum_{q,q_1\le \sqrt{X}}\sum_{\max(q^2,q_1^2)\le y\le X}B\left(\dfrac{y}{q}\right)B\left(\dfrac{y}{q_1}\right)\ll \sum_{q\le \sqrt{X}}\left|\sum_{q^2\le y\le X}B\left(\dfrac{y}{q}\right)\right| \left|\sum_{q_1\le \sqrt{X}}B\left(\dfrac{y}{q_1}\right)\right|$$ Thus, it should be as desired that $$\int_{0}^X R(y)^2 dy\ll \sum_{q\le \sqrt{X}} q\cdot \sqrt{X}\ll X^{3/2}.$$