I'm struggling with the integral, $$\int\frac{1}{(x-2)^4 \sqrt{x^2 + 6x + 2}}dx.$$
I tried it as follows:
Substituting $x-2 = \frac1t \implies dx = \frac{-dt}{t^2}.$ $$\therefore \int\frac{dx}{(x-2)^4 \sqrt{x^2 + 6x + 2}} = \int \frac{- dt}{\frac{t^2}{t^4} \sqrt{(\frac1t + 2)^2 + 6 (\frac1t + 2) + 2}} = \int \frac{-t^3}{\sqrt{18t^2 + 10t + 1}}\ dt$$
How to continue from here?
On
Continuing where you left off,
$$\begin{align*} I &= - \int \frac{t^3}{\sqrt{18t^2+10t+1}} \, dt \\[1ex] &= -16 \int \frac{(s-5)^3}{(s^2-18)^4} \, ds \end{align*}$$
by employing the substitution,
$$s = \frac{\sqrt{18t^2+10t+1}-1}t \implies t = \frac{2s-10}{18-s^2} \implies dt = \frac{2(18-10s+s^2)}{(18-s^2)^2} \, ds$$
The rest can be done by expanding into partial fractions.
Alternatively, starting over,
$$\begin{align*} I &= \int \frac{dx}{(x-2)^4 \sqrt{x^2 + 6x + 2}} \\[1ex] &= \frac18 \int \frac{(1-y^2)^3}{\left(y^2+\sqrt2\,y-4\right)^4} \, dy \tag{1} \\[1ex] &= -\frac1{4\sqrt2} \int \frac{(z^2-2)^3}{\left(z^2+2z-8\right)^4} \, dz \tag{2} \end{align*}$$
and again with partial fractions.
$$y = \frac{\sqrt{x^2+6x+2} - \sqrt2}{x} \implies x=\frac{2\sqrt2\,y-6}{1-y^2} \implies dx=\frac{2\left(\sqrt2\,y^2-6y+\sqrt2\right)}{(1-y^2)^2} \, dy$$
I'm going to continue from the last known position above. In the end, you'll get a polynomial in secants, which just results in using standard tricks from calculus II (along with integration by parts).
Let's start with the standard secant substitution $t + \frac{5}{18} = \sqrt{\frac{7}{18}} \sec y,$ so $dt = \sqrt{\frac{7}{18}} \sec y \tan y \ dy.$ Our integral becomes
$$ \int \frac{-t^3}{\sqrt{18t^2 + 10t + 1}}\ dt = -\frac{1}{\sqrt{18}} \int \frac{ \left( \sqrt{\frac{7}{18}} \sec y - \frac{5}{18} \right)^3 \left( \sqrt{\frac{7}{18}} \sec y \tan y \ dy \right)}{\sqrt{\frac{7}{18}} \tan y}$$
and thankfully the stuff in the denominator cancels. This leaves
$$ -\frac{1}{\sqrt{18}} \int \left( \sqrt{\frac{7}{18}} \sec y - \frac{5}{18} \right)^3 \sec y \ dy. $$
It gets quite messy, but after expanding the binomial, we get
$$ -\frac{1}{\sqrt{18}} \int a^3 \sec^4 y - 3a^2 b \sec^3 y + 3ab^2 \sec^2 y -b^3 \sec y \ dy$$
where I've relabeled $a = \sqrt{\frac{7}{18}}$ and $b = \frac{5}{18}$ to make things cleaner. The first term is handled with the substitution $z = \tan y$ (after pulling two secants - again, standard calculus II technique), the second using integration by parts (choose $u = \sec y, \ dv = \sec^2 y \ dy$), and the third and fourth are standard results. I won't flesh out all the details here, but we obtain:
$$ -\frac{1}{\sqrt{18}} \left( \frac{a^3 \tan^3 y}{3} + a^3\tan y - \frac{3a^2 b}{2} \left( \ln |\sec y + \tan y| + \sec y \ \tan y \right) \\ + 3ab^2 \tan y - b^3 \ln |\sec y + \tan y| \right) + C $$
(wow!). We're almost home free. By drawing a triangle based on our secant substitution, we obtain $\tan y = \sqrt{ \frac{1}{a^2} \left( t - b \right)^2 - 1}.$ Finally, invert your initial substitution to get $t = \frac{1}{x-2}.$
Now, by being lazy (sorry!), we arrive at our result:
$$-\frac{1}{\sqrt{18}} \frac{\left( a \sqrt{ \frac{1}{a^2} \left( \frac{1}{x-2} - b \right)^2 - 1} \right)^3}{3} + a^3 \left(\sqrt{ \frac{1}{a^2} \left( \frac{1}{x-2} - b \right)^2 - 1} \right) - \frac{3a^2 b}{2} \left( \ln \left( \frac{1}{a} \left(\frac{1}{x-2} - b \right) + \left( \sqrt{ \frac{1}{a^2} \left( \frac{1}{x-2} - b \right)^2 - 1} \right) \right) + \left( \sqrt{ \frac{1}{a^2} \left( \frac{1}{x-2} - b \right)^2 - 1} \right) \frac{1}{a} \left(\frac{1}{x-2} - b \right) \right) + 3b^2 \left(\frac{1}{x-2} - b \right) - b^3 \ln \left| \left( \frac{1}{a} \left(\frac{1}{x-2} - b \right) + \left( \sqrt{ \frac{1}{a^2} \left( \frac{1}{x-2} - b \right)^2 - 1} \right) \right) \right| + C$$
Wow! I may have mismatched some parentheses here, but that's a result.