I am trying to find the first 5 terms of the solution of: $$x^2y''-2xy'+ln(x)y=0$$ using the power series method (around $x_0=1$). I don't know how many terms should I keep from each infinite sum (when testing the power series to the equation), especially when I need to multiply the Taylor series of $lnx$ with $y$. Is it possible to find the n-th term $a_n$, where $\sum_{n=0}^\infty a_n(x-1)^n$ ?
This is a related question, but I am not sure whether one can find $a_n$ here.
Let's start by supposing that $$y(x)=\sum_{n=0}^\infty a_n(x-1)^n.\tag{0}$$ Then we see that:
\begin{eqnarray}x^2y'' &=& x^2\sum_{n=0}^\infty\frac{d^2}{dx^2}\left[a_n(x-1)^n\right]\\ &=& x^2\sum_{n=0}^\infty n(n-1)a_n(x-1)^{n-2}\\ &=& x^2\sum_{n=2}^\infty n(n-1)a_n(x-1)^{n-2}\\ &=& \left((x-1)^2+2(x-1)+1\right)\sum_{n=2}^\infty n(n-1)a_n(x-1)^{n-2}\\ &=& \sum_{n=2}^\infty n(n-1)a_n(x-1)^n+\sum_{n=2}^\infty 2n(n-1)a_n(x-1)^{n-1}+\sum_{n=2}^\infty n(n-1)a_n(x-1)^{n-2}\\ &=& \sum_{n=0}^\infty n(n-1)a_n(x-1)^n+\sum_{n=1}^\infty 2n(n-1)a_n(x-1)^{n-1}+\sum_{n=2}^\infty n(n-1)a_n(x-1)^{n-2}\\ &=& \sum_{n=0}^\infty n(n-1)a_n(x-1)^n+\sum_{n=0}^\infty 2(n+1)na_{n+1}(x-1)^n+\sum_{n=0}^\infty (n+2)(n+1)a_{n+2}(x-1)^n\\ &=& \sum_{n=0}^\infty \bigl[n(n-1)a_n+2(n+1)na_{n+1}+(n+2)(n+1)a_{n+2}\bigr](x-1)^n\\ &=& \sum_{n=0}^\infty \Bigl[\bigl(n^2-n\bigr)a_n+\bigl(2n^2+2n\bigr)a_{n+1}+\bigl(n^2+3n+2\bigr)a_{n+2}\Bigr](x-1)^n\end{eqnarray}
and
\begin{eqnarray}2xy' &=& 2x\sum_{n=0}^\infty\frac{d}{dx}\left[a_n(x-1)^n\right]\\ &=& 2x\sum_{n=0}^\infty na_n(x-1)^{n-1}\\ &=& 2x\sum_{n=1}^\infty na_n(x-1)^{n-1}\\ &=& \bigl(2(x-1)+2\bigr)\sum_{n=1}^\infty na_n(x-1)^{n-1}\\ &=& \sum_{n=1}^\infty 2na_n(x-1)^n+\sum_{n=1}^\infty 2na_n(x-1)^{n-1}\\ &=& \sum_{n=0}^\infty 2na_n(x-1)^n+\sum_{n=1}^\infty 2na_n(x-1)^{n-1}\\ &=& \sum_{n=0}^\infty 2na_n(x-1)^n+\sum_{n=0}^\infty 2(n+1)a_{n+1}(x-1)^n\\ &=& \sum_{n=0}^\infty \bigl[2na_n+2(n+1)a_{n+1}\bigr](x-1)^n\\ &=& \sum_{n=0}^\infty \bigl[2na_n+(2n+2)a_{n+1}\bigr](x-1)^n.\end{eqnarray} Thus, we find that $$x^2y''-2xy'=\sum_{n=0}^\infty\Bigl[\bigl(n^2-3n\bigr)a_n+\bigl(2n^2-2\bigr)a_{n+1}+\bigl(n^2+3n+2\bigr)a_{n+2}\Bigr](x-1)^n.\tag{1}$$
Also, note that for any $n\ge 0$ we have $$a_n(x-1)^n\ln(x)=a_n(x-1)^n\sum_{m=1}^\infty\frac{(-1)^{m-1}}m(x-1)^m=\sum_{m=1}^\infty\frac{(-1)^{m-1}a_n}m(x-1)^{m+n}.$$ For any given non-negative integer $j,$ we will have $m+n=j$ if and only if $m\le j$ and $n=j-m.$ Since each $m>0,$ this will be impossible for $j=0.$ For $j=1,$ it is possible if and only if $(m,n)=(1,0).$ For $j=2,$ it is possible if and only if $(m,n)=(1,1)$ or $(m,n)=(2,0).$ For $j=3,$ it is possible if and only if $(m,n)=(1,2)$ or $(m,n)=(2,1)$ or $(m,n)=(3,0).$ In general, for a given $j,$ it is possible if and only if $(m,n)=(1+k,j-1-k)$ for some integer $k$ such that $0\le k\le j-1.$ This allows us to rewrite
\begin{eqnarray}\ln(x)y &=& \sum_{n=0}^\infty\sum_{m=1}^\infty\frac{(-1)^{m-1}a_n}m(x-1)^{m+n}\\ &=& \sum_{j=0}^\infty\left(\sum_{k=0}^{j-1}\frac{(-1)^{(1+k)-1}a_{j-1-k}}{1+k}\right)(x-1)^{(1+k)+(j-1-k)}\\ &=& \sum_{j=0}^\infty\left(\sum_{k=0}^{j-1}\frac{(-1)^ka_{j-1-k}}{1+k}\right)(x-1)^j,\end{eqnarray}
and reindexing by $i=j-1-k$ (so that $k=j-1-i$), we get $$\ln(x)y=\sum_{j=0}^\infty\left[\sum_{i=0}^{j-1}\frac{(-1)^{j-1-i}a_i}{j-i}\right](x-1)^j,$$ or equivalently, $$\ln(x)y=\sum_{j=0}^\infty\left[-\sum_{i=0}^{j-1}\frac{(-1)^{j-i}a_i}{j-i}\right](x-1)^j.\tag{2}$$
Using $(1)$ and $(2),$ we can rewrite $0=x^2y''-2xy'+\ln(x)y$ in the form $$0=\sum_{j=0}^\infty b_j(x-1)^j,$$ and if we want this to be true anywhere other than $x=1,$ we want $0=b_j$ for all $j\ge 0.$ Thus, for all $j\ge 0,$ we require that $$0=\bigl(j^2-3j\bigr)a_j+\bigl(2j^2-2\bigr)a_{j+1}+\bigl(j^2+3j+2\bigr)a_{j+2}-\sum_{i=0}^{j-1}\frac{(-1)^{j-i}a_i}{j-i},$$ or $$0=j(j-3)a_j+2(j+1)(j-1)a_{j+1}+(j+2)(j+1)a_{j+2}-\sum_{i=0}^{j-1}\frac{(-1)^{j-i}a_i}{j-i},$$ which we can rewrite in the recursive form $$a_{j+2}=-\frac{j(j-3)}{(j+2)(j+1)}a_j-\frac{2(j-1)}{j+2}a_{j+1}+\frac{1}{(j+2)(j+1)}\sum_{i=0}^{j-1}\frac{(-1)^{j-i}a_i}{j-i}.\tag{$\star$}$$ This might look intimidating to solve, and if we needed to find all the terms, it might be! Let's dig into it for the first several $j$ to see what we can find out.
Case $j=0$: Here, $(\star)$ becomes $$a_2=0a_0+a_1+0=a_1.\tag{3}$$
Case $j=1$: Here, $(\star)$ becomes $$a_3=\frac{2}{6}a_1+0a_2+\frac{1}{6}\cdot\frac{(-1)^1a_0}{1}=\frac13a_1-\frac16a_0.\tag{4}$$
Case $j=2$: Here, $(\star)$ becomes \begin{eqnarray}a_4 &=& \frac{2}{12}a_2-\frac{2}{4}a_3+\frac{1}{12}\cdot\frac{(-1)^2a_0}{2}+\frac{1}{12}\cdot\frac{(-1)^1a_1}{1}\\ &=& \frac{1}{6}a_2-\frac{1}{2}a_3+\frac{1}{24}a_0-\frac{1}{12}a_1\\ &\overset{(4)}{=}& \frac{1}{6}a_2-\frac{1}{6}a_1+\frac{1}{12}a_0+\frac{1}{24}a_0-\frac{1}{12}a_1\\ &=& \frac{1}{6}a_2-\frac{1}{4}a_1+\frac{1}{8}a_0\\ &\overset{(3)}{=}& \frac{1}{6}a_1-\frac{1}{4}a_1+\frac{1}{8}a_0,\end{eqnarray} and so $$a_4=-\frac{1}{12}a_1+\frac18a_0.\tag{5}$$
Continuing in this manner for the cases $j=3,4,5,...$ will allow us to determine as many $a_j$ as we like, each of which should be in terms of the unknown constants $a_0.$ and $a_1.$ It's even possible (though I haven't verified it) that we might be able to determine a closed form for $a_j$ with $j\ge 2$ in terms of $a_0$ and $a_1,$ which could then provide us a closed form for $y(x).$
I leave the rest to you! Please let me know if you have any questions about how/why I did anything above, if you have difficulties continuing, or even if you just want to verify your results. Welcome to Math.SE!