I need to decompose this fraction: $${x^2+1\over (x-1)^3\cdot(x+3)}$$
I tried to write it up like this: $${A\over (x-1)}+{B\over (x-1)^2}+{C\over (x-1)^3}+{D\over (x+3)}$$ But now i get $$A\cdot(x-1)^5\cdot (x+3) +B\cdot (x-1)^4\cdot (x+3) +C\cdot(x-1)^3\cdot (x+3) +D\cdot (x-1)^6 =x^2+1$$ which is not correct i think.
What should i do?
Your first step is correct, in fact to find the partial fraction decomposition the first thing you need to do is:
$$\dfrac{x^2+1}{(x-1)^3 (x+3)} = \dfrac{A}{x-1} + \dfrac{B}{(x-1)^2} + \dfrac{C}{(x-1)^3} + \dfrac{D}{x+3}$$
Now what you need to do is to sum the fractions on the left side in this way:
$$\dfrac{x^2+1}{(x-1)^3 (x+3)} = \dfrac{A}{x-1} + \dfrac{B}{(x-1)^2} + \dfrac{C}{(x-1)^3} + \dfrac{D}{x+3} $$
$$\iff \dfrac{x^2+1}{(x-1)^3 (x+3)}= \dfrac{A(x-1)^2(x+3) + B(x-1)(x+3) + C(x+3) + D(x-1)^3}{(x-1)^3(x+3)}$$
$$\iff x^2+1 = A(x-1)^2(x+3) + B(x-1)(x+3) + C(x+3) + D(x-1)^3$$
Now you can expand the polynomial $(x-1)^2 (x+3)$, $(x-1)(x+3)$, $(x-1)^3$ and you should get that the left hand side is equal to
$$Ax^3 +Ax^2-5Ax+3A + Bx^2+2Bx-3B+Cx+3C+Dx^3 -3Dx^2 +3Dx -D$$
now you can compare the coefficient, for example you see that $x^2+1$ has degree $2$ and hence you can conclude that $Ax^3 + Dx^3 = 0$. If you do this reasoning for $x^2, x, \text{and the constants}$ you should get a linear system like this
$$\begin{pmatrix}1 & 0 & 0 & 1 \\ 1 & 1 & 0 & -3 \\ -5 & 2 & 1 & 3 \\ 3 & -3 & 3 & -1 \end{pmatrix} \begin{pmatrix} A \\ B \\ C \\ D\end{pmatrix} = \begin{pmatrix} 0 \\ 1 \\ 0 \\ 1 \end{pmatrix}$$
The solutions are given by
$$\begin{pmatrix}A \\B\\C\\D\end{pmatrix} = \begin{pmatrix} \frac{5}{32} \\ \frac{3}{8} \\ \frac{1}{2} \\ \frac{-5}{32}\end{pmatrix}$$
and therefore you partial fraction decomposition is
$$\dfrac{x^2+1}{(x-1)^3 (x+3)} = \dfrac{5}{32(x-1)} + \dfrac{3}{8(x-1)^2} + \dfrac{1}{2(x-1)^3} - \dfrac{5}{32(x+3)}$$