I have to do the problem through the Sine/Cosine formulation of Fourier Series, so I'm talking about those coefficients. The interval is [-π, π].
I did the problem and checked it via Wolfram Alpha and confirmed my result. But it doesn't make sense! Since $Cos$3$(x)$ is an even function, only the $a$o and $a$n coefficients should be present, and the $b$k coefficients will vanish. I also found that the $a$o coefficient vanishes. So all that should be left would be the $a$k terms.....But in the solution I got for these coefficients, there is a $Sin(k$π$)$ term! This term will be zero for all k = 1, 2, 3,...
So apparently ALL the coefficients vanish... That cannot be true, and I don't know what could be wrong. I'm confident in my calculation since Wolfram got the exact same result.
Wolfram Alpha's result is not well defined when $k=1$ or $k=3$ (you get a 0/0 form), which are where the contributions turn out to be. In this case a different recipe than the one Wolfram Alpha is using is required for the integral.
The easy way to derive the Fourier coefficients in this case is not by integration but by direct trigonometry. This is easier in complex variables: $\cos(t)^3=\left ( \frac{e^{it}+e^{-it}}{2} \right )^3 = \frac{e^{3it}+3e^{it}+3e^{-it}+e^{-3it}}{8}=\cos(3t)/4+3\cos(t)/4$.
Another way is to use the expression you wrote, which is valid when $k$ is not $-3,-1,1$ or $3$, and take the limit as $k$ tends to $1$ or $3$. Since you have a 0/0 form, L'Hopital's rule or a similar technique is required to compute this limit.