I have a function $y(x) = y_1(x) + y_2(x)$ consisting of two other waveforms, where
$ y_1(x) = \cos{\left(\dfrac{16 \pi}{5} x \right)}; \, y_2(x) = \displaystyle \sum_{k=-\infty}^{\infty} y_3(x - k); \, y_3(x) = \begin{cases} x & \text{if } 0 < x < 1 \\ 0 & \text{otherwise} \end{cases} \tag*{} $
In other words, $y_2(x)$ is a sawtooth of period 1, amplitude 1, that starts rising at $x = 0$. The period of $y(x)$ is $T = \text{LCM} (\frac{5}{8}, 1) = 5 $. In the following image, $y(x)$ is shown in blue, $y_1(x)$ in yellow, and $y_2(x)$ in green:
Both by hand and using Mathematica I found the Fourier coefficients of $y(x)$, obtaining the trigonometric form:
$ y(x) = \dfrac{1}{2} - \displaystyle \sum_{n=1}^\infty \left[ \dfrac{1 + \cos{(\frac{2 \pi n}{5})} + \cos{(\frac{4 \pi n}{5})} + \cos{(\frac{6 \pi n}{5})} + \cos{(\frac{8 \pi n}{5})}}{\pi n} \sin{\left( \frac{2 \pi n}{5} x \right)} \right] \tag*{} $
or the complex/exponential form:
$ y(x) = \displaystyle \sum_{n=-\infty}^\infty \left[ i \dfrac{(-1)^n \left( (-1)^n + \left( 2 \cos{\frac{\pi n}{5}} + \cos{\frac{3 \pi n}{5}} \right) \right)}{2 \pi n} \exp{\left(i \frac{2 \pi n}{5} x \right)} \right] \tag*{} $
These computations are shown in Mathematica in the following image:
But when I plot any of the two previous expressions, they don't look like the original $y(x)$. In the following image, the original expression for $y(x)$ is shown in blue, while its trigonometric Fourier series approximation (up to the 30th harmonic) is shown in yellow:
For some reason, the Fourier series looks like the sawtooth. What did I do wrong?
Edit: Computation of $a_n$
Following Olivier's answer, I got:
$\begin{align} a_n &= \dfrac{2}{T} \displaystyle\int_0^T y(x) \cos{(n \omega_0 x)} \, \mathrm dx \\ &= \dfrac{2}{5} \displaystyle\int_0^5 \left( y_1(x) + y_2(x) \right) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx \\ &= \dfrac{2}{5} \displaystyle\int_0^5 \left( \cos{\left(\dfrac{16 \pi}{5} x \right)} \cos{(\dfrac{2 \pi n}{5} x)} + \cos{\left(\dfrac{2 \pi n}{5} x\right)} \displaystyle \sum_{k=-\infty}^{\infty} y_3(x - k) \right) \, \mathrm dx \\ &= \dfrac{2}{5} \left( \underbrace{\displaystyle\int_0^5 \cos{\left(\dfrac{16 \pi}{5} x \right)} \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx}_{I_1} + \cdots \right. \\ & \left. \cdots + \underbrace{\displaystyle\int_0^5 (y_3(x) + y_3(x - 1) + y_3(x - 2) + y_3(x - 3) + y_3(x - 4)) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx}_{I_2} \right) \end{align}$
Using a table of common integrals, for $I_1$ I got:
$ I_1 = \begin{cases} 0 & \text{if } n \ne 8 \\ \dfrac{5}{2} & \text{if } n = 8 \end{cases} $
For $I_2$:
$I_2 = \displaystyle\int_0^1 x \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx + \displaystyle\int_1^2 (x - 1) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx + \displaystyle\int_2^3 (x - 2) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx + \cdots$ $\cdots + \displaystyle\int_3^4 (x - 3) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx + \displaystyle\int_4^5 (x - 4) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx$
The five integrals of $I_2$ have the general form (where $m$ is an integer):
$\displaystyle\int_m^{m+1} (x - m) \cos{\left(\dfrac{2 \pi n}{5} x\right)} \, \mathrm dx = \dfrac{25}{4 \pi^2 n^2} \cos{\left(\dfrac{2 \pi n}{5} [m+1]\right)} + \cdots $ $\cdots + \dfrac{5}{2 \pi n} [m+1] \sin{\left(\dfrac{2 \pi n}{5} [m+1]\right)} - \dfrac{25}{4 \pi^2 n^2} \cos{\left(\dfrac{2 \pi n}{5} [m]\right)} - \dfrac{5}{2 \pi n} [m] \sin{\left(\dfrac{2 \pi n}{5} [m]\right)} - \cdots $ $\cdots - \dfrac{5}{2 \pi n} [m+1] \sin{\left(\dfrac{2 \pi n}{5} [m+1]\right)} + \dfrac{5}{2 \pi n} [m] \sin{\left(\dfrac{2 \pi n}{5} [m]\right)} $
Subtituting this integral for $m=0,1,2,3,4$ in $I_2$ and simplifying:
$I_2 = 0 $
Subtituting $I_1$ and $I_2$ in $a_n$ and simplifying:
$ a_n = \begin{cases} 0 & \text{if } n \ne 8 \\ 1 & \text{if } n = 8 \end{cases} $
I updated the trigonometric Fourier series (shown in yellow) to include this and plotted it, and now it looks like the original $y(x)$ (shown in blue):
I don't understand why Mathematica failed initially to compute $a_n$. And I'm ashamed of needing help for this.
I find it suspicious that $a_n=0$ since the function $y-\dfrac{1}{2}$ is not odd. According to your calculations $y-\dfrac{1}{2}$ is a pure sinus wave.
You should check the calculations of $a_n$ when $n=8$, the integral now involves $$\cos^2\left(\dfrac{16\pi t}{5}\right)$$ and is unlikely to vanish.