Problem
For what values of $a$ and $b$ is
$$\int \frac{ax+b}{(x-1)(x+1)^2}dx$$
a rational function?
Attempted solution:
At first I was a bit stumped since this integral superficially appears to be a rational function for any real $a$ and $b$ since it might typically lead to some kind of logarithm and an arctangent.
However, I decided to try partial fractions to see if this gets me anywhere. Perhaps figuring out the integral will give me a clear decision on what to do for $a$ and $b$. Perhaps selecting certain values will cancel out something and let the integral end up being a rational function when otherwise it will not.
$$\int \frac{ax+b}{(x-1)(x+1)^2}dx = \int \frac{A}{x-1} + \frac{B}{x+1} + \frac{C}{(x+1)^2}dx $$
$$A(x+1)^2 + B(x^2-1) + C(x-1) = $$ $$A(x^2+2x+1) +Bx^2 -B +Cx -C =$$ $$Ax^2 + 2Ax +A + Bx^2 -B +Cx -C$$ $$x^2(A +B) + x(2A+C) +A -B -C$$
This leads to
$$A + B = 0$$ $$2A + C = a$$ $$A-B-C = b$$
However, this is not solvable because it has too many unknowns.
My second attempt was to simplify the integral and see if I can still integrate without using partial fractions:
$$\int \frac{ax+b}{(x-1)(x+1)^2}dx = a \int\frac{x}{(x-1)(x+1)^2}dx + b\int \frac{1}{(x-1)(x+1)^2} dx$$
At this point, I do not see any obvious substitutions that could work. Partial integration is also not obvious to me how it could work. Perhaps some tricky trigonometric substitution, but multiple trails run cold and lead nowhere.
The expected answer is all $a$ and $b$ where $a = -b$
Where are some productive ways to solve this problem?
Your partial fraction approach was correct. You're looking for the decomposition $$\frac{ax+b}{(x-1)(x+1)^2}=\frac{A}{x-1}+\frac{B}{x+1}+\frac{C}{(x+1)^2}$$ which implies $ax+b = A(x+1)^2+B(x-1)(x+1)+C(x-1)$. The unknowns here are $A, B, C$. Little $a$ and $b$ are parameters. In addition to obtaining a system and solving it, you can plug in $x=\pm1$ to find the values (also known as the cover-up method). So $x=1$ gives $A=\frac{a+b}{4}$ and $x=-1$ gives $C=\frac{a-b}{2}$. To find $B$, notice you have $(A+B)x^2=0x^2$, so $B=-A=-\frac{a+b}{4}$. The integral is then $$A\ln|x-1|+B\ln|x+1|-\frac{C}{x+1}+\text{constant}$$ which is a rational function when the logarithms vanish, i.e., $A=B=0 \Leftrightarrow a+b = 0 \Leftrightarrow a=-b$.