Let $f(x)$ be a function, then for its Fourier series $$ f(x) = \frac{a_0}{2} + \sum_{n=1}^{\infty} (a_n \cos(nx) + b_n \sin(nx)) $$ I found two different definitions (both yielding different series).
i) I found \begin{align*} a_n & = \frac{1}{\pi} \int_0^{2\pi} f(x) \cos(nx) d x \quad n \ge 0, \\ b_n & = \frac{1}{\pi} \int_0^{2\pi} f(x) \sin(nx) d x \quad n > 0 \end{align*} and ii) \begin{align*} a_n & = \frac{1}{\pi} \int_{-\pi}^{\pi} f(x) \cos(n x) d x \\ b_n & = \frac{1}{\pi} \int_{-\pi}^{\pi} f(x) \sin(n x) d x. \end{align*} If I evaluate $f(x) = x(1-x)$ according to ii), then I get $$ a_n = (-1)^n \frac{4\pi}{n^2} \quad \mbox{ and } \quad b_n = (-1)^{n+1} \frac{2}{n} $$ when I use i) I get $$ a_n = \frac{-1}{\pi^2 n^2} \quad \mbox{ and } \quad b_n = 0. $$ Both are different, but I guess the Fourier series is unique, or are they both "different" Fourier series, can someone explain why it is not unique and why there are both definitions around (and what is the better one?) or what is their relation?
You are evaluating Fourier series for two different functions.
For your first case, we suppose $f(x)=x(1-x)\, x\in[0,2\pi]$ is given and extended it to a $2\pi$ periodic function.
While for your second case, we suppose $f(x)=x(1-x)\, x\in[-\pi,\pi]$ is given and extended it to a $2\pi$ periodic function. Note after extension, these are two different functions, say on $[-\pi,0)$. That's why you get two different Fourier series.
There is no called "better one" among these two definitions. It depends on how your function is given. If the function's information is given on $[a,b]$, you just use the definition on that interval to calculate the coefficients.