The assignment is to solve: $$ \int_{0}^{\infty}f(x) dx $$ where $$f(x) = \frac{4}{(x+1)^2(x+3)} $$ .
I did partial fraction on the indefinite integral of $f(x)$ and got:
$$
f(x) = \frac{A}{x+1} + \frac{B}{(x+1)^2} + \frac{C}{x+3}
$$
$$
4=(A+C)x^3 + (5A+B+3C)x^2 + (7A+4B+3C)x + (3A + 3B + C)
$$
and put the last three functions inside a matrix:
$$
\begin{bmatrix}
5 & 1 & 3 & 0 \\
7 & 4 & 3 & 0 \\
3 & 3 & 1 & 4 \\
\end{bmatrix}
$$
gauss elimination:
\begin{bmatrix}
1 & 0 & 0 & -9 \\
0 & 1 & 0 & 6 \\
0 & 0 & 1 & 13 \\
\end{bmatrix}
Replace $A$, $B$ and $C$ with the result in $f(x)$:
$$ f(x) = \frac{-9}{x+1} + \frac{6}{(x+1)^2} + \frac{3}{x+3} $$
Calculate the definite integral from 0 to $\infty$:
$$ \lim_{b \to \infty} \int_{0}^{b} f(x)dx = \lim_{b \to \infty} (-9 \log|b+1|- \frac{6}{b+1} + 13 \log|b+3| + 6 - 13\log(3)) $$
Separate into several limits: $$ \lim_{b \to \infty}(-9 \log|b+1|- \frac{6}{b+1} + 13 \log|b+3|) + 6 - 13\log(3) $$
L'Hopital: $$ \lim_{b \to \infty}(\frac{9}{b+1} + \frac{6}{(b+1)^2} + \frac{13}{b+3}) + 6 - 13\log(3) $$
All limits are zero, so we're just left with the constants: $$ 6 - 13\log(3) $$
which is not the correct answer.
Your mistake was that this equation
$$4=(A+C)x^3 + (5A+B+3C)x^2 + (7A+4B+3C)x + (3A + 3B + C)$$
was wrong. You start from
$$\frac4{(x+1)^2(x+3)}=\frac{A}{x+1}+\frac{B}{(x+1)^2}+\frac{C}{x+3}$$
Hence, assuming that $x\neq-1$ or $x\neq-3$, you have that
$$4=A(x+1)(x+3)+B(x+3)+C(x+1)^2\\ \iff (A+C)x^2+(4A+B+2C)x+3A+3B+C=4$$
Thus the system to be solved is
$$A+C=0,\quad 4A+B+2C=0,\quad 3A+3B+C=4$$
what gives the solutions $A=-1,\, B=2$ and $C=1$.