This is a problem given to me by fractals on Art of Problem Solving. I couldn't solve it so I'm posting it here for some thoughts on it.
Let $$S = \sum_{j = 0}^{\lfloor n/2 \rfloor} \sin\dfrac{2\pi\cdot j^2}{n}$$
Then prove that if $n \equiv 0,3 \pmod{4}$ we have $S = \frac{\sqrt{n}}{2}$ and if $n \equiv 1,2 \pmod{4}$ we have $S = 0$. This problem can be somewhar reduced via $\sin \theta = \frac{e^{i \theta} - e^{-i \theta}}{2i}$. For the case of $n$ prime this turns the problem into the classical evaluation of a Quadratic Gauss Sum, so for $n$ prime the problem is trivial. However, when $n$ is not prime the formula still seems to hold based on numerically doing small cases. The proof for primes I know fails utterly in generalizing (squaring $S$ doesn't tell us much anymore), so idea would be appreciated.
Since this seems a rather natural question to ask, I would assume almost surely there is some paper published on this before. However, searching I couldn't find anything I wanted unfortunately.
You can find a proof in "Analytic Number Theory" by H. Iwaniec, p50
Let $G(m)$ be the Gauss sum of type $$ G(m)=\sum_{n\textrm{ (mod $m$)}} e\left(\frac{n^2}{m}\right) $$ where $e(x)=e^{2\pi i x}$.
Then we have $$ G(m)=\frac{1+i^{-m}}{1+i^{-1}}\sqrt{m}. $$
There is a little cleaning up to do, to get to the sum that you wrote, (taking imaginary part and taking care of extra term)