How do I express $cos(3\theta)$ and $sin^{2}(\theta)$ in Legendre's polynomials, knowing that $x=cos\theta$?
I know that $f(x)=\sum a_{n}P_{n}(x)$ and $P_{n}=\frac{(-1)^{n}}{2^{n}n!}\frac{d^{n}}{dx^{n}}(1-x^{2})^{n}$, but I don't know what to do with them
If the functions you are trying to match to Legendre polynomials of $\cos \theta$ are sinusoids, the easiest thing to do is usually to use trigonometric identities to re-express the function in terms of $\cos \theta$. Then start from the highest order, match up the coefficients, and work your way down.
For example, suppose $f(\theta) = \cos(4 \theta) - \sin^2 (\theta)$. We have $$ \cos(4 \theta) = 2 \cos^2(2 \theta) - 1 = 2 (2 \cos (\theta) - 1)^2 - 1 = 8 \cos^4 \theta - 8 \cos^2 \theta + 1 = 8x^4 - 8x^2 + 1 $$ and $\sin^2(\theta) = 1 - \cos^2 \theta = 1 - x^2.$ So $$ f(x) = 8 x^4 - 7 x^2. $$ The highest order of $x$ is 4, so the highest Legendre polynomial present is $P_4(x) = \frac{1}{8}(35 x^4 - 30 x^2 + 3).$ Subtract that out: $$ f(x) - \frac{64}{35} P_4(x) = 8 x^4 - 7 x^2 - \frac{8}{35} (35 x^4 - 30 x^2 + 3) = - \frac{1}{7} x^2 - \frac{24}{35} $$ Now we're down to $x^2$, so we can add in a multiple of $P_2(x)$: $$ f(x) - \frac{64}{35} P_4(x) + \frac{2}{21} P_2(x) = - \frac{1}{7} x^2 - \frac{24}{35} + \frac{1}{21} (3 x^2 - 1) = - \frac{11}{35} $$ And $P_0(x) = 1$, so $$ f(x) - \frac{64}{35} P_4(x) + \frac{2}{21} P_2(x) + \frac{11}{15} P_0(x) = 0 \\ f(x) = \frac{64}{35} P_4(x) - \frac{2}{21} P_2(x) - \frac{11}{15} P_0(x). $$