Why is $\text{Li}(x)$ a much better estimate of $\pi(x)$ than $\frac{x}{\log x}$?

Question

I'm reading Prime Obsession by John Derbyshire, and quite central to the thesis of the book is the PNT. I understand the concept of the logarithmic distribution of primes (i.e. the probability that a number in the neighborhood of $x$ is prime is approximately $\frac{1}{\log x}$). If you think of this as a "prime density function," then we can say that $\pi(x) \sim \frac{x}{\log x}$.

Now comes $\text{Li}(x) = \int_0^x \frac{1}{logt} dt$. Derbyshire says on page 115 that "[$\text{Li}(x)$] is a much better estimate" of $\pi(x)$ than $\frac{x}{logx}$.

On a conceptual level, why is this true? I have a feeling that it's kind of like how work is better "estimated" by $\int_0^d F(t) dt$ than by $F * d$, but I can't put my finger on exactly what's going on (what would the analog of $F$ be in this context? How does this explain the fact that $\frac{x}{\log x}$ is always an underestimate? etc.).

I have an understanding of calculus that goes up to introductory multivariable calculus, but both a layman-ish explanation and a rigorous/mathematical explanation would be appreciated.

Answer 1

The key idea is the abscissa of convergence of Dirichlet series and Mellin transforms.

• $\zeta(s) = s\int_1^\infty \lfloor x \rfloor x^{-s-1}dx =\frac{s}{s-1}+s\int_1^\infty (\lfloor x \rfloor-x) x^{-s-1}dx$ thus $\log (\zeta(s)(s-1))$ is analytic at $s=1$.

  Let $\Pi(x) = \sum_{p^k \le x} \frac{1}{k}= \pi(x)+\mathcal{O}(x^{1/2})$ then the Euler product and Abel summation formula give $\log \zeta(s) = s \int_2^\infty \Pi(x) x^{-s-1}dx$

• Integrating by parts shows $s\int_2^\infty \text{Li}(x)x^{-s-1}dx+\log(s-1)$ is analytic at $s=1$, where $\text{Li}(x) = \int_2^x \frac{t}{\log t}$

  $s\int_2^\infty \frac{x}{\log x}x^{-s-1}dx+\log(s-1)$ is bounded but not analytic at $s=1$.

• $\int_2^\infty (\Pi(x)-\text{Li}(x))x^{-s-1}$ is analytic at $s=1$. Together with the zero-free region $\sigma > 1-\frac{A}{1+|\log t|}$ and the estimate $\log (\zeta(s)(s-1)) = \mathcal{O}(\log^k |t|)$ and Mellin inversion, it gives the prime number theorem : for every $m$, $\Pi(x)-\text{Li}(x) = o(\frac{x}{\log^m x})$.

  Under the Riemann hypothesis $\Pi(x)-\text{Li}(x) = \mathcal{O}(x^{1/2}\log x)$.

  With $\frac{x}{\log x}$ you'll only get $\Pi(x)-\frac{x}{\log x} = \mathcal{O}(\frac{x}{\log^2 x})$.

Answer 2

Not an answer but a comment which needed some space

A table for some value of $n$

$ \begin{array}{r|r|r|r} n & \text{Li}(n) &\dfrac{n}{\log{n}}&\pi(n)\\ \hline 1000000 & 78628 & 72383 & 78498 \\ 2000000 & 149055 & 137849 & 148933 \\ 3000000 & 216971 & 201152 & 216816 \\ 4000000 & 283353 & 263127 & 283146 \\ 5000000 & 348639 & 324151 & 348513 \\ 6000000 & 413077 & 384437 & 412849 \\ 7000000 & 476827 & 444123 & 476648 \\ 8000000 & 540000 & 503305 & 539777 \\ 9000000 & 602677 & 562053 & 602489 \\ 10000000 & 664919 & 620421 & 664579 \\ 100000000 & 5762210 & 5428682 & 5761455 \\ 200000000 & 11079975 & 10463629 & 11078937 \\ 300000000 & 16253410 & 15369410 & 16252325 \\ 400000000 & 21337379 & 20194906 & 21336326 \\ 500000000 & 26356833 & 24962409 & 26355867 \\ 600000000 & 31326046 & 29684689 & 31324703 \\ 700000000 & 36254243 & 34370014 & 36252931 \\ 800000000 & 41147863 & 39024158 & 41146179 \\ 900000000 & 46011649 & 43651380 & 46009215 \\ 1000000000 & 50849235 & 48254943 & 50847534 \\ 10000000000 & 455055615 & 434294482 & 455052511 \\ 20000000000 & 882214880 & 843205936 & 882206716 \\ 30000000000 & 1300016132 & 1243550986 & 1300005926 \\ 40000000000 & 1711964132 & 1638528672 & 1711955433 \\ 50000000000 & 2119666540 & 2029608840 & 2119654578 \\ 60000000000 & 2524048318 & 2417638082 & 2524038155 \\ 70000000000 & 2925709820 & 2803166336 & 2925699539 \\ 80000000000 & 3325071593 & 3186579089 & 3325059246 \\ 90000000000 & 3722444262 & 3568161225 & 3722428991 \\ 100000000000 & 4118066401 & 3948131654 & 4118054813 \\ \end{array} $

Answer 3

This is because $x/\log x$ always under estimates $\pi(x)$ for all $x > 10$; i.e. we can show that $\pi(x) > x/\log x$. On the other hand, $\pi(x) - Li(x)$ changes sign infinitely many times i.e. $Li(x)$ never under estimates or over estimates $\pi(x)$ continuously.

This is because the actual statement and proof of the prime number theorem says that $\pi(x) \sim Li(x)$. Note that the prime number theorem never actually says that $\pi(x) \sim x/\log x$ is that best we can do. However because $Li(x) \sim x/\log x$ we tend to simplify the original statement of the prime number theorem and say that $\pi(x) \sim x/\log x$ but because of this simplification, we lose the accuracy in approximating $\pi(x)$ due to which $x/\log x$ ends us under estimating $\pi(x)$.