This solution I have found in my book for the indefinite integral
$\int{\frac{1}{\sqrt{x^2-9}}dx}=ln|x+\sqrt{x^2-9}| +c$
I think that this is acceptable only for $0<\theta<\frac{\pi}{2}$.
But what about when $\frac{\pi}{2}<\theta<\pi$ ? Is it the same? I think it is opposite to the above, means:
$\int{\frac{1}{\sqrt{x^2-9}}dx}= -ln|x+\sqrt{x^2-9}| +c$.
beacuse:
$\sqrt{x^2-9}=\sqrt{\tan^2\theta}=|\tan\theta|$
which is positive when $0<\theta<\frac{\pi}{2}$ and negative when $\frac{\pi}{2}<\theta<\pi$.
Can you explain me what happens in this situation?
How do I work out the integral?
Actually, when you are asked to compute an indefinite integral of a function, you should be told which is the interval of definition. If not, things like that can happen.
As you know, if we take $f(x)=\frac1x$ in $(0,+\infty)$, then $\int f(x) dx=\ln x + C$, but if we define $f$ in $(-\infty, 0)$, then we would obtain $\int f(x) dx=\ln(-x)+C$.
In your case, I assume you are using the substitution $x=3\sec \theta$, with $x\in[-3,3]$, so $\theta\in[0,\pi]$. And $\tan \theta<0$ when $\theta\in[\frac\pi2, \pi]$. So it does depend on the interval, as you said. But in this case, I think you're supposed to forget about it and assume $\sqrt{\tan^2\theta}=\tan \theta$.