So, I know that the coefficients for Fourier transformation to the function: $x^2-\pi x +\frac{\pi^2}{6}$ should be $a_0 = 0$, $a_n = \frac{1}{n^2}$ and $b_n = 0$. The $a_0$ is correctly computated, but I am trying to integrate for the $a_n$, and I also get $0$. By partially integrating, I get to integrate $\cos$ which evaluates to 0 at $\pi$ or to compute the upper limits (in this case $\pi$ of $\sin $ and it again evaluates to 0. Where is my mistake?
$$a_0 = \frac{1}{\pi}\int^{\pi}_0 (x^2-\pi x +\frac{\pi^2}{6})dx = \frac{1}{\pi}\left[ \left.\frac{x^3}{3}-\frac{\pi x^2}{2}+\frac{\pi^2 x}{6} \right|^{\pi}_0\right] = \frac{1}{\pi}\left(\frac{2\pi^3 -3\pi^3+\pi^3}{6}\right) = 0 $$
$$a_n = \frac{2}{\pi}\left[\int^{\pi}_{0}x^2 \cos(2nx)dx -\int^{\pi}_0\pi x \cos(2nx)dx +\frac{\pi^2}{6}\cos(2nx) dx \right] = \frac{2}{\pi} \left.\left[\frac{1}{2n}x^2 \sin(2nx)\right|^{\pi}_0 - \\ 2\int^{\pi}_0 x\cos(2nx) dx -\left(\left. \frac{\pi x}{2n}\sin(2nx)\right|^{\pi}_0 - \int^{\pi}_0 \pi\cos(2nx)dx \right) +\left. \frac{\pi^2}{12n}\sin(2nx)\right|^{\pi}_0\right] = 0$$
$\int_0^{\pi} x^{2} \cos (2nx)dx =\frac 1 {2n} x^{2} \sin (2nx)|_0^{\pi} -\frac 1 n \int_0^{\pi} x\sin (2nx)dx$. You have $\cos (2nx)$ instead of $\sin (2nx)$ in this last integral.