So, I've been doing this problem over and over for a while. I want to get it right and I figure I've almost got it. However, I don't understand why the answer is what it is.
Anyway this is the integral, and its answer:
$$ \int \frac{\sqrt{x^2 - 9}}{x^3} dx = \frac{\sqrt{x^2-9}}{2x^2} - \sin^{-1}{\frac{x}{3}} + C$$
I don't understand where the first rational came from, or the inverse sine. I have an idea of how those things work, but I just don't know how the solution led to that. Anyway, this is what I tried:
Trigonometric substitution:
$$ x = 3\sec{u} \\ dx = 3\sec{u}\tan{u} du $$
Anyway, this integral is long, I ended up at this point:
$$\frac{1}{6}[u - \sin{u}\cos{u}] + c $$
I think it is correct at that point, but I'm not sure. I'm not sure why the answer ends up looking like it does above. I was wondering if someone could help me out.
$$I= \int \frac{\sqrt{x^2 - 9}}{x^3}\ dx$$
Let $x=\sec u$ as you did
$dx=\sec u \tan u du$
$$I= \int \frac{3\tan^2 u}{\sec^3u}\ du $$
$$\frac{I}{3}= \int \cos u\ du -\int \cos^3u\ du $$
Can you finish?
As OP ended up with $$\frac{1}{6}(u - \sin{u}\cos{u}) + C $$ where $u=\sec^{-1}(x)$
Construct a triangle:

$$\sin u=\frac{\sqrt{x^2-1}}{x} \implies u=\sin^{-1}(\frac{\sqrt{x^2-1}}{x}) $$
You can manipulate it to any inverse function you want :)