I would like to calculate the following sum, which looks like a Gauss sum.
Let $n$ be a natural number and let $a,b$ be integers. Denote by $e(x)=e^{2\pi i x/n}$. Consider the sum $$ \sum_{1 \leq j, k \leq n} e(aj^2+2jk+bk^2).$$
I tried to calculate this sum by reducing this to the normal Gauss sum as follows. By completing square the exponent, we have $$aj^2+2jk+bk^2=a(j+\frac{k}{a})^2+(b-\frac{1}{a})k^2.$$ Thus the sum is equal to $$\sum_{k=1}^ne \left((b-\frac{1}{a})k^2 \right)\sum_{j=1}^n e\left(a(j+\frac{k}{a})^2\right).$$ Now this looks pretty much like a quadratic Gauss sum except that $j+\frac{k}{a}$ and $b-\frac{1}{a}$ might not be integers.
Could you give me advise how to calculate this sum?
Assume $\gcd(a,n)=1$. Define $c$ by $ac\equiv1\bmod n$. Then $$aj^2+2jk+bk^2=a(j^2+2cjk+bck^2)=a((j+ck)^2+(bc-c^2)k^2)$$ so the sum is $$\sum_ke(a(bc-c^2)k^2)\sum_je(a(j+ck)^2)$$ This solves your problem about things not being integers.