I'm trying to obtain a recursion formula relating $a_{n+2}$ to $a_n$ using the method of undetermined coefficients. Then determining $a_n$ explicitly for each $n$ and finding the sum of the series.
$$(1-x^2)y''-2xy'+6y=0$$
Taking the first and second derivative of the power series gives me:
$$(1)\space y = \sum_{n=0}^{\infty}a_nx^n $$ $$(2)y'\space = \sum_{n=1}^{\infty} na_nx^{n-1}$$ $$(3) \space y'' = \sum_{n=2}^{\infty}n(n-1)a_nx^{n-2}$$
Then plugging this into the differential equation:
$$ (a) \space \sum_{n=2}^{\infty}n(n-1)a_nx^{n-2}+\sum_{n=2}^{\infty}n(n-1)a_nx^{n}-2\sum_{n=1}^{\infty} na_nx^{n} + 6 \sum_{n=0}^{\infty}a_nx^n$$
Then simplifying the exponents: $$(b) \space \sum_{n=0}^{\infty}(n+2)(n+1)a_{n+2}x^{n} + \sum_{n=0}^{\infty}n(n-1)a_{n}x^{n} - 2\sum_{n=0}^{\infty} na_nx^{n} + 6 \sum_{n=0}^{\infty}a_nx^n $$
Then simplifying with $x^n$:
$$(c) \space \left([(n+2)(n+1)a_{n+2}] + [n(n-1-2)+6]a_n\right)x^n$$
Finally, equating coefficients of $x^n$ to obtain the relation:
$$(d) \space [(n+2)(n+1)a_{n+2}] + [n(n-1-2)+6]a_n=0$$
Or what amounts to: $$(4) \space a_{n+2} = \frac{n(n-1-2)a_n}{(n+2)(n+1)}$$ $$(5) \space a_n = \frac{(n+2)(n+1)}{n(n-1-2)}$$
I would really appreciate a check on my calculations and whether I missed something out or went wrong along the way. Additionally, how do I proceed from here?
This is a special case of Legendre's differential equation at $\ell=2$.
The series expansion should be $$\sum_{n=2}^{\infty}n(n-1)a_nx^{n-2}\color{blue}-\sum_{n=2}^{\infty}n(n-1)a_nx^{n}-2\sum_{n=1}^{\infty} na_nx^{n} + 6 \sum_{n=0}^{\infty}a_nx^n=0$$ from which $$(n+2)(n+1)a_{n+2}-(n+3)(n-2)a_n=0.$$ Note that $a_2=-3a_0$ and $12a_4=0$ so $a_{2N}=0$ for all $N>1$. Iterating gives \begin{align}a_{n+2}&=\frac{(n+3)(n-2)}{(n+2)(n+1)}a_n=\frac{(n+3)(n-2)}{(n+2)(n+1)}\frac{(n+1)(n-4)}{n(n-1)}a_{n-2}=\cdots\\&=(n+3)(n+1)\frac{(n-1)!(n-2k)(n-2k-2)}{(n-2k+1)!}\frac{(n-2k)!}{(n+2)!}a_{n-2k}\\&=\frac{n+3}{(n+2)n}\frac{(n-2k)(n-2k-2)}{n-2k+1}a_{n-2k}.\end{align} When $n=2N+1$ choosing $k=N$ gives $$a_{2N+3}=-\frac{N+2}{(2N+3)(2N+1)}a_1$$ so the solution is $$y(x)=a_0\left(1-3x^2\right)+a_1x\left(1-\frac23x^2-\frac15x^4-\frac4{35}x^6-\cdots\right).$$ Indeed, taking $a_0=1/2$ and $a_1=0$ yields the Legendre polynomial of order two.