I'm trying to proof something right now. What exactly is not important. I was able to break down the whole thing to the following equation.
$ \sum_{j=1}^{M/2}\sum_{k=1}^{M-1}\cos{\left(\frac{2\pi}{M}kj\right)}=-M/2 $
This should hold for all even values of $M\in\mathbb{N}$. Any ideas? I tried to substitute both summation by a single one, but that yields to a big mess....
I checked it in Mathematica with different values of $M$.
Change to exponential notation. Let $j$ be fixed and let $d=\gcd(j,M).$ Then $$\sum_{k=1}^{M-1} \exp(2\pi i j k/M)$$ adds up some $M$-th roots of unity. If you were to increase the upper limit to $M$, then you'd be adding all the $M/d$-th roots of unity $d$ times, which adds to $0$. So: $$\sum_{k=1}^{M-1} \cos(2\pi j k/M) = \Re\left(\sum_{k=1}^{M-1} \exp(2\pi i j k/M)\right)$$ $$= \Re\left(\sum_{k=1}^{M} \exp(2\pi i j k/M) -\exp(2\pi i jM/M)\right) = \Re(0 - 1) = -1.$$ And that should make the outer sum easy.