Modular transformations of $\eta(\tau)$

Question

Under a modular transformation the Dedekind $\eta$ function transforms as $$\eta(-1/\tau) = \sqrt{-i \tau}\eta(\tau) \, .\tag*{$(*)$}$$Siegel gives a proof in this paper here that uses complex analytic techniques. However, I am wondering how we can see it in the following fashion. The Dedekind $\eta$ function can be represented in the form$$\eta(\tau) = q^{1/24} \prod_{n = 1}^\infty (1 - q^n) = \sum_{n = -\infty}^\infty (-1)^n q^{{3\over2}(n - 1/6)^2}.$$How can we use the Poisson resummation formula and this representation of the $\eta$ function to verify the modular transformation $(*)$?

Answer 1

Let us start from one side of what we want to prove,$$\eta(-1/\tau) = r^{1/24} \prod_{n = 1}^\infty (1 - r^n), \quad r = \exp\left(-{{2\pi i}\over\tau}\right).$$We were given Euler's (quite remarkable) pentagonal number theorem, so we might as well start by using it to transform the product into a sum$$\eta(-1/\tau) = r^{1/24} \sum_{n \in \mathbb{Z}} (-1)^n r^{(3n^2 - n)/2}, \quad r = \exp\left(-{{2\pi i}\over\tau}\right).$$The other hammer we were given is the Poisson summation, and this expression is beginning to look like the right nail: we just need to a find a function $f$ such that each term in the sum above is $f(n)$, and then Fourier transform it. Rewriting the above expression slightly as$$f(n) = \exp\left[\pi i\left(-{1\over{12\tau}} + n - {{(3n^2 - n)}\over\tau}\right)\right],$$it becomes straightforward to extrapolate it to the reals by simply replacing $n$ with a real number $x$. We can now find the Fourier transform (note that $f$ does fall off "sufficiently fast" because $\text{Im}\,\tau > 0$),\begin{align*} \tilde{f}(k) & = \int_{-\infty}^\infty dx\,\exp\left[\pi i\left(-{1\over{12\tau}} + x - 2kx - {{(3x^2 - x)}\over\tau}\right)\right] \\ & = \sqrt{\tau\over{3i}} \exp\left[\pi i\left({{\tau(-2k + 1 + {1\over\tau})^2}\over{12}} - {1\over{12\tau}}\right)\right] \\ & = \sqrt{{-i\tau}\over3}\exp\left[\pi i\left({{\tau(2k - 1)^2}\over{12}} - {{(2k - 1)}\over6}\right)\right],\end{align*}where in the first step we used the Gaussian integral$$\int_{-\infty}^\infty dx\,\exp(-ax^2 + bx + c) = \sqrt{\pi\over a} \exp\left({{b^2}\over{4a}} + c\right),$$valid for $\text{Re}\,a > 0$, which in our case translates to$$\text{Re}\left({{3\pi i}\over\tau}\right) = \text{Im}\left(-{{3\pi}\over\tau}\right) > 0.$$We can finally use the Poisson summation formula to obtain$$\eta(-1/\tau) = \sqrt{{-i\tau}\over3} \sum_{k \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(2k - 1)^2}\over{12}} - {{(2k - 1)}\over6}\right)\right].\tag*{$(*)$}$$How close is this to the desired result? Let us use the pentagonal number theorem on $\eta(\tau)$ as well to transform it into a sum$$\eta(\tau) = q^{1/24} \sum_{n \in \mathbb{Z}} (-1)^n q^{(3n^2 - n)/2} = \sum_{n \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(6n - 1)^2}\over{12}} + n\right)\right].$$This is close! However, there is a factor of $\sqrt{3}$, and the term proportional to $\tau$ in the exponent has $6n - 1$ instead of $2k - 1$. To solve these assorted factors of $3$, we can split the sum in $(*)$ into three sums, with $k = 3l$, $3l + 1$, $3l + 2$. Then$$\eta(-1/\tau) = \sqrt{{{-i\tau}\over3}}\sum_{l \in \mathbb{Z}} \left[ \exp\left[\pi i\left({{\tau(6l - 1)^2}\over{12}} - {{(6l - 1)}\over6}\right)\right] + \exp\left[\pi i\left({{\tau(6l + 1)^2}\over{12}} - {{(6l + 1)}\over6}\right)\right] + \exp\left[\pi i\left({{\tau(6l + 3)^2}\over{12}} - {{(6l + 3)}\over6}\right)\right]\right].$$The first term looks like the desired expression times $e^{\pi i/6}$. In the second term, we can take $l \to -l$ to transform it into the desired expression times $e^{-\pi i/6}$. Since$$e^{\pi i/6} + e^{-\pi i/6} = 2\cos \pi/6 = \sqrt{3},$$we have found the full answer and the last term must vanish!

We can rewrite it as a sum over odd integers,$$\sum_{m \in 2\mathbb{Z} + 1} \exp\left[\pi i\left({{3\tau m^2}\over4} - {m\over2}\right)\right],$$and now separate the positive and negative $m$ parts to find$$\sum_{m \in 2\mathbb{Z} + 1,\,m > 0} \exp\left[\pi i{{3\tau m^2}\over4}\right]\left(e^{\pi im/2} + e^{-\pi im/2}\right),$$which vanishes since $e^{\pi im} = -1$ for $m$ an odd integer.

Therefore, we finally have$$\eta(-1/\tau) = \sqrt{-i\tau} \sum_{l \in \mathbb{Z}} \exp\left[\pi i\left({{\tau(6l - 1)^2}\over{12}} - l\right)\right] = \sqrt{-i\tau}\eta(\tau),$$as we wanted to show.