I thought it would be this : $$2/π\int_{0}^{π} x^2\cos(nx) dx = 2/π\int_{0}^{π} x^2(-1)^n = 2/π(-1)^n\int_{0}^{π} x^2=\frac{2}{π(-1)^n}\biggl[\frac{x^3}{3}\biggr]_0^π =\frac{2(-1)^n}{3π^3}. $$
But it is actually $$\frac{2}{π}\int_{0}^{π} x^2\cos(nx) dx = \frac{4(-1)^n}{n^2}$$ according to my professor's notes. How did he get that answer?
Thanks.
You cannot say that $\cos nx = (-1)^n$. This is valid for $x=\pi$. A counterexample would be $\cos n\cdot0 = 1$, and this doesn't depend on $n$.
The correct way to do this integral is using integration by parts, multiple times.
Hint: $u=x^2$, $\text dv = \cos nx\ \text dx$.