Well, I am checking this out because even though I know the problem (a 2nd order differential) can be solved more easily, I want to try this out.
So we have $10 \cos t$ and want the Fourier transform. $\cos t$ is an even function, which means it goes to zero on any symmetric interval. But we can prove this if we have to because $a_0 = \frac{1}{\pi} \int^{\pi}_{-\pi} 10 \cos t \ dt$ and that turns into $\frac{10}{\pi}(\sin t)$ from $\pi$ to $-\pi$ and that's zero. So there goes $a_0$.
Meanwhile, $a_n$ goes to zero too, because $a_n = \frac{2}{\pi} \int^{\pi}_{0}10 \cos t \cos nt \ dt$. I don't want to mess with integration by parts so I pull out the trig identity and get $$a_n = \frac{20}{\pi} \int^{\pi}_{0}\frac{1}{2} [\cos (t(1-n)) + \cos (t(1+n))] \ dt$$ which gets me $$a_n = \frac{10}{\pi} \left[ \frac{\sin(t(1-n))}{1-n}+\frac{\sin(t(1+n))}{1+n}\right]^{\pi}_{0}$$
and that gets me to zero, because any multiple of $\pi$ in a sin function goes to zero.
On to the other coefficient, $b_n$. For that one I use: $$b_n = \frac{2}{\pi} \int^{\pi}_{0}10 \cos t \sin nt \ dt$$ and using the trig identities again to make the integration easier I get $$\frac{5}{\pi} \int^{\pi}_{0}\frac{1}{2} [\sin (t(1+n)) - \sin (t(1-n))] \ dt$$ and integrating that I should get $$= \frac{5}{\pi} \left[ \frac{-\cos(t(1+n))}{1+n}+\frac{\cos(t(1-n))}{1-n}\right]^{\pi}_{-\pi}$$
and it's here where I think I have a mistake. Because the final bit I got was $$\frac{5}{\pi} \left[\frac{2(-1)^{n+1}}{1-n}\right]$$
and I know I should get $10 \cos t$ as my series so there's an algebra mistake in here someplace.
Any ideas would be helpful here. I'm sure it's something obvious.
There are two minor mistakes:
Mistake 1: Your integration formulas work only when $n \neq 1$. When $n=1$ you have a 0 in the denominator!
You have to treat the case $n=1$ separately.
Mistake 2 $\left[ \frac{-\cos(t(1+n))}{1+n}+\frac{\cos(t(1-n))}{1-n}\right]$ is periodic with period $2\pi$, thus your final bit has to be $0$. You made a mistake when you calculated it. The mistake you made is that
$$\frac{\cos(\pi(1-n))}{1-n} - \frac{\cos((-\pi)(1-n))}{1-n}=0 \,,$$ not $\frac{2(-1)^{n+1}}{1-n}$. Note that $\cos$ is even.
Also note that the function you integrate in this part is odd, the integral from $-\pi$ to $\pi$ has to be $0$.