I have a piecewise function:
$$ f(t) = \begin{cases} 7 & 0 < t < \pi \\ -7 & \pi < t < 2\pi \end{cases} $$
and it's assumed that when f (t) is extended to the negative t-axis in a periodic manner, the resulting function is even. Therefore, the particular solution is a cosine series.
The differential equation given is: $$ 8 \frac{d^2x}{dt^2} + 9x = f(t) $$
The particular solution is in the form of: $$ x_p(t) = \sum_{n=1}^{\infty} A_n \ \cos{\frac{n\pi t}{p}} $$
Problem: I am trying to retrieve $A_n$.
I tried to find the particular solution using the method of undetermined coefficients:
$$ x_c (t) = c_1\cos{\frac{3}{2\sqrt{2}}t} + c_2\sin{\frac{3}{2\sqrt{2}}t}\\ x_p (t) = C \cdot A_n \cos{\frac{n\pi t}{2 \pi}} $$
Using $C \cdot A_n \cos{\frac{nt}{2}}$, differentiating and plugging into the DE:
$$ 8(\frac{-Cn^2}{4} \cdot A_n \cos{\frac{nt}{2}}) + 9(C \cdot A_n \cos{\frac{nt}{2}}) = C \cdot A_n \cos{\frac{nt}{2}} \\ C = (9-2n^2) $$
Finding $A_n$ using Fourier Cosine series and @Dylan's answer: $$ \frac{1}{\pi} \int_{0}^{\pi} 7 \cos{\frac{n}{2}t} \ dt + \frac{1}{\pi} \int_{\pi}^{2\pi} -7 \cos{\frac{n}{2}t} \ dt = 28 \frac{\sin{\frac{n}{2}n\pi}}{n\pi} $$
Using $C = (9-2n^2)$: $$ \sum_{n=1}^{\infty} A_n (9-2n^2) \cos{\frac{n}{2}t} = \sum_{n=1}^{\infty} 28 \frac{\sin{\frac{n}{2}\pi}}{n\pi} \cos{\frac{n}{2}t}\\ A_n = 28 \frac{\sin{\frac{n}{2}\pi}}{n\pi(9-2n^2)} $$
You should have noticed that $\sin(n\pi)=0$. This is the first sign that your answer is wrong.
Since $f(t)$ is an even function, its period is $4\pi$, not $2\pi$. Hence, $f(t)$ is represented accordingly as
$$ f(t) = A_0 + \sum_{n=1}^\infty A_n \cos\left( \frac{n}{2}t \right) $$
Integration should be over $[0,2\pi]$. You have two separate integrals to do here: one for the constant term and one for $n\ne 0$
\begin{align} A_0 &= \frac{\int_0^{2\pi} f(t)\ dt}{\int_0^{2\pi} 1\ dt} = \frac{1}{2\pi}\int_0^{2\pi} f(t)\ dt \\ A_n &= \frac{\int_0^{2\pi} f(t)\cos\left( \frac{n}{2}t \right) dt }{\int_0^{2\pi} \cos^2 \left(\frac{n}{2}t\right)\ dt} = \frac{1}{\pi} \int_0^{2\pi} f(t)\cos\left( \frac{n}{2}t \right) dt \end{align}
You'll need to break them up into integrals over $[0,\pi]$ and $[\pi,2\pi]$, since $f(t)$ is piece-wise
Once you have the Fourier series of $f(t)$, you can make a guess for $x_p$ of the form
$$ x_p(t) = c_0 + \sum_{n=1}^\infty c_n\cos\left(\frac{n}{2}t\right) $$
Note that this expression corresponds to $p=2\pi$ in your given formula. Then
$$ 8\ddot{x}_p + 9x_p = 9c_0 + \sum_{n=1}^\infty (9-2n^2)c_n\cos\left(\frac{n}{2}t\right) $$
Comparing coefficients gives
\begin{align} 9c_0 &= A_0 \\ (9-2n^2)c_n &= A_n \end{align}
In this example, you conveniently have $A_0=0$. But this is not true for the general case, so the constant term should always be considered.