Let f(x) be the fractional part of x. I'm asked to find $F(x)=f*f=\int_{0}^{1}f(x-t)f(t)dt$ explicitly and find its Fourier series.
My first idea was to just integrate $frac(x-t)frac(t)$, getting $\frac{frac(x)}{2}-1/3$. However, I know that the Fourier coefficients of F must be the square of the coefficients of the fractional part and hence my result is wrong.
Thinking about how (x-t) can be negative, my next attempt was to split the integral (with no success).
So how should I proceed?
By the Fourier series of the sawtooth wave we have $$\{x\}=\frac{1}{2}-\frac{1}{\pi}\sum_{k\geq 1}\frac{\sin(2\pi k x)}{k}$$ uniformly over any compact subset of $\mathbb{R}\setminus\mathbb{Z}$. For $x\in(0,1)$
$$ \int_{0}^{1}\{x-t\}\{t\}\,dt = \int_{0}^{x}(x-t)t\,dt +\int_{x}^{1}(1+x-t)t\,dt = \frac{1}{6}+\frac{x}{2}-\frac{x^2}{2} $$ and $$F(x)=\frac{1}{4}-\frac{1}{2\pi^2}\sum_{k\geq 1}\frac{\cos(2\pi k x)}{k^2}.$$