$(0 < x < \pi) : f(x) = x(\pi - x)$ and I need to prove that $$f(x) = \frac{\pi^2}{6} - \left(\frac{\cos(2x)}{1^2} + \frac{\cos(4x)}{1^2} + \frac{\cos(6x)}{1^3} ...\right)$$ using Fourier series.
I've been trying this problem for 3 days now, I'm getting a correct series but it's different from the series I am to proof. and what I tried so far is:
let $$g( x) =\begin{cases} f( x) & 0< x< \pi \\ f( -x) & -\pi < x< 0 \end{cases}$$
now $g$ is an even function so $b_n = 0$ and $a_0 = 1/\pi \int_{-\pi}^{\pi}g(x)\mathrm{d}x$ and $ a_n = 1/\pi\int _{-\pi}^{\pi} g(x)\cos(nx)\mathrm{d}x$
$$a_0 = \frac{1}{\pi} \left(\int_{-\pi}^{0} -x(\pi+x)\mathrm{d}x + \int_{0}^{\pi}x(\pi-x)\mathrm{d}x\right) = \frac{\pi^2}{3}$$
$$a_n = \frac{1}{\pi}\left(\int_{-\pi}^{0}-x(\pi+x)\cos(nx)\mathrm{d}x +\int_{0}^{\pi}x(\pi-x)\cos(nx)\mathrm{d}x\right)$$
$$a_n\pi= -\pi\int_{-\pi}^{0}x\cos(nx)\mathrm{d}x - \int_{-\pi}^{0}x^2\cos(nx)\mathrm{d}x + \pi\int_{0}^{\pi}x\cos(nx)\mathrm{d}x - \int_{0}^{\pi}x^2\cos(nx)\mathrm{d}x$$
$x\cos(nx)$ is odd function and $x^2\cos(nx)$ is even function. so the above equation can be reduced as
$$a_n\pi= 2\pi\int_{0}^{\pi}x\cos(nx)\mathrm{d}x - 2\int_{0}^{\pi}x^2\cos(nx)\mathrm{d}x$$
$$\frac{a_n\pi}{2}= \int_{0}^{\pi}(x\pi - x^2)\cos(nx)\mathrm{d}x$$
$$\frac{a_n\pi}{2}= \bigg[(x\pi - x^2)\sin(nx)/n\bigg]_{0}^{\pi} - 1/n\int_{0}^{\pi}\sin(nx)(\pi - 2x)\mathrm{d}x$$
$$\frac{a_n\pi}{2}=0 - 1/n\left(\bigg[-(\pi-2x)\cos(nx)/n\bigg]_{0}^{\pi} - 2/n\int_{0}^{\pi}\cos(nx)\mathrm{d}x \right)$$ Since $\cos(n \pi) = (-1)^n$,
$$\frac{a_n\pi}{2}= - 1/n\left(\left(\frac{\pi}{n}((-1)^n+1)\right) - 2/n\cdot 0 \right)$$
$$a_n = \frac{-2}{n^2} ((-1)^n+1)$$
substituting $a_0, a_n$ in the series
$$g(x) = \frac{a_0}{2} + \sum_{n=1}^{\infty}a_n\cos(nx)$$
$$g(x) = \frac{\pi^2}{6} + \sum_{n=1}^{\infty} \frac{-2}{n^2} ((-1)^n+1) \cos(nx)$$
$$g(x) = \frac{\pi^2}{6} + \frac{-2}{2^2} \cos(2x) + \frac{-2}{4^2} \cos(4x) + \frac{-2}{6^2} \cos(6x) + ...$$
I can confirm that the above answer will converge to to $f(x)$ here on Desmos
Your second-to-last step is correct; your error is in the last step. The quantity $(-1)^n + 1$ is equal to 0 for odd $n$, but is equal to 2 for even $n$ (not 1, as you appear to have assumed). Thus, the terms in your sum are \begin{align*} -\frac{4}{2^2} &\cos (2x)- \frac{4}{4^2} \cos (4x)- \frac{4}{6^2} \cos (6x)+ \dots \\ &= - \frac{2^2}{(2\cdot 1)^2} \cos (2x)- \frac{2^2}{(2\cdot 2)^2} \cos (4x)- \frac{2^2}{(2\cdot 3)^2}\cos (6x)+ \dots \\ &=-\frac{1}{1^2} \cos (2x)- \frac{1}{2^2} \cos (4x)- \frac{1}{3^2} \cos (6x)+ \dots \end{align*} as desired.