If ${F}_{n} \left(z\right) = \prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)$ show that the series \begin{align}f \left(z\right) = - \sum_{n = 2}^{\infty} \frac{{F}_{n} \left(z {n}^{- 1}\right)}{\left({z}^{n} {n}^{- n} - 1\right) {n}^{n - 1}}\end{align} is analytic except at roots of the equations ${z}^{n} = {n}^{n}$ and that the sum of the residues of $f \left(z\right)$ over an annular region centered at the origin with inner radius $n$ and outer radius $n + 1$ counts the number of primes less than $n + 1$.
Attempt: Proving that $f \left(z\right)$ is analytic involves a simplification of the summand.
\begin{align} f \left(z\right) & = - \sum_{n = 2}^{\infty} \frac{{F}_{n} \left(z {n}^{- 1}\right)}{\left({z}^{n} {n}^{- n} - 1\right) {n}^{n - 1}} \\ & = - \sum_{n = 2}^{\infty} \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)}{\left({z}^{n} {n}^{- n} - 1\right) {n}^{n - 1}} \\ & = - \sum_{n = 2}^{\infty} n \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)}{{z}^{n} - {n}^{n}} \\ \end{align}
Now it is apparent from the form of the equation that $f \left(z\right)$ is a meromorphic function that is analytic everywhere except at any of the $n$th roots of a natural number.
These roots shall be $n {e}^{k \frac{2 \pi i}{n}}$ where $k = 0 , \ldots , n - 1$ after defining the branch cut to be along the positive real numbers.
\begin{align} {R}_{n {k}_{0}} = & {\text{Res}}_{z = n {e}^{{k}_{0} \frac{2 \pi i}{n}}} \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)}{{z}^{n} - {n}^{n}} \\ = & {\text{Res}}_{z = n {e}^{{k}_{0} \frac{2 \pi i}{n}}} \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)}{\prod_{m = 0}^{n - 1} \left(z - n {e}^{m \frac{2 \pi i}{n}}\right)} \\ = & \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {z}^{k l}\right)}{\prod_{m = 0 , m \ne {k}_{0}}^{n - 1} \left(z - n {e}^{m \frac{2 \pi i}{n}}\right)} {|}_{z = n {e}^{{k}_{0} \frac{2 \pi i}{n}}} \\ = & \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {n}^{k l} {e}^{{k}_{0} k l \frac{2 \pi i}{n}}\right)}{{n}^{n - 1} \prod_{m = 0 , m \ne {k}_{0}}^{n - 1} \left({e}^{{k}_{0} \frac{2 \pi i}{n}} - {e}^{m \frac{2 \pi i}{n}}\right)} \\ = & \frac{\prod_{k = 1}^{n - 1} \prod_{l = 1}^{n - 1} \left(1 - {n}^{k l} {e}^{{k}_{0} k l \frac{2 \pi i}{n}}\right)}{{n}^{n - 1} {e}^{2 \pi i {k}_{0} \frac{n - 1}{n}} \prod_{m = 0 , m \ne {k}_{0}}^{n - 1} \left(1 - {e}^{\left(m - {k}_{0}\right) \frac{2 \pi i}{n}}\right)} \\ \end{align}
Question: I am having trouble proving that $\sum_{n ' = 2}^{n} \sum_{{k}_{0} = 0}^{n ' - 1} {R}_{n ' {k}_{0}} = {\chi}_{P}$ where $\chi$ is the characteristic function and $P$ is the set of primes. I believe that I must use number theory to prove this result, but I am not advanced enough to realize the specific means of proof. I have tried examining the special case when $n$ is a prime number, but I do not realize any means to simplify the above products.
Note: I feel that this question is very similar to the 2015 A3 Putnam problem.
It is very important that the numerator is actually $F_n(z/n)$: in calculating the residues, you have used $F_n(z)$. Furthermore, we have to multiply all residues by a factor of $n$, since that is also present in the series’ numerator. We must also multiply by $-1$.
A simple roots-of-unity fact: $$\prod_{m=1}^{n-1}\left(1-\exp\frac{2\pi i m}{n}\right)=n$$
Note that as $m$ varies, avoiding $k_0$, $(m-k_0)$ touches every residue class mod $n$, so the product on the denominator is exactly the product I mention above. This is an important principle I use later. Replacing the product on the denominator by $n$, the residue calculation then looks like this:
$$\frac{\color{red}{-n}}{n^ne^{2\pi i k_0/n}}\prod_{k,l=1}^{n-1}(1-(ne^{k_0\cdot2\pi i/n}\color{red}{/n})^{kl})=-\frac{e^{-2\pi i k_0/n}}{n^{n-1}}\prod_{k,l=1}^{n-1}(1-e^{2\pi i k_0kl/n})$$
Suppose $k_0\neq0$ for now. If $n$ is prime, $n$ divides $k_0kl$ iff. it divides one of them. However, they are all less than $n$ and nonzero, so this is not possible. Furthermore, for fixed $k_0,k$, we have that $k_0kl\equiv k_0kl’$, mod $n$, iff. $n$ divides $k_0k(l-l’)$; due to the small nature of each term, and primality of $n$, that happens only when $l=l’$. So, the product in $e^{k_0kl\cdot2\pi i/n}$ will, as $l$ varies, touch every $n$th root of unity except $1$. Using our simple fact above, that makes the inner $l$-product equal to $n$. Then $\prod_{k=1}^{n-1}n=n^{n-1}$, and the residue looks like: $$R_{n,k_0}=-e^{-2\pi i k_0/n}$$After a cancellation.
If $k_0=0$ then the residue term is zero since you face $(1-1)$ in the product.
If $n$ isn’t prime, then it is a product of two numbers $a,b$, both less than $n$. That means, for any choice of $k_0$, I can find the pair $(k,l)=(a,b)$, and get $(1-1)=0$ in the product from $2\pi i k_0kl/n=2\pi i k_0$. Hence, that residue term will be zero.
Then: $$\begin{align}\sum_{n’=2}^n\sum_{k_0=0}^{n’-1}R_{n’,k_0}&=0+\sum_{n’=2\\n’\text{ prime}}^n\sum_{k_0=1}^{n-1}(-e^{-2\pi i k_0/n’})\\&=\sum_{n’=2\\n’\text{ prime}}^n1\\&=\pi(n)\end{align}$$
As desired!
Note: in the problem text, you only want to sum over the residues in the annulus between $n$ and $n+1$, so I’m not sure if summing $\sum_{n’=1}^n[...]$ is really what you want. But, it’s what you wrote at the bottom of your question so it’s what I addressed - although, it isn’t equal to $\chi_P(n)$, but equal to $\pi(n)$. What is equal to $\chi_P(n)$, is the internal sum - taking residues within the described annulus - $\sum_{k_0=0}^{n-1}R_{n,k_0}$. So, it’s not fully clear to me what the problem statement actually is - but I am confident my calculation is correct, and I hope it helps you with what you need.