I used integration by parts: let u=$\sec^{-1}\,x$, dv=dx, then du=$\frac{1}{|x|\sqrt{x^2-1}}$, v=x.
I = $x\sec^{-1}\,x\;-\;\int\frac{x}{|x|\sqrt{x^2-1}}dx\\$
Integration of $\int\frac{x}{|x|\sqrt{x^2-1}}dx$:
Let x=$\sec\theta$, then dx = $\sec\theta\tan\theta\,d\theta$. $\theta\in(0, \frac{\pi}2)\cup(\frac{\pi}2, \pi)\\$.
$\int\frac{x}{|x|\sqrt{x^2-1}}dx\,=\,\int\frac{\sec\theta}{|\sec\theta|\sqrt{\sec^2\theta-1}}\sec\theta\tan\theta\,d\theta\\\qquad\qquad\qquad=\,\int\frac{\sec\theta}{|\sec\theta|}\frac{\tan\theta}{|\tan\theta|}\sec\theta\,d\theta$
When $\theta\in(0, \frac{\pi}2),\;\sec\theta=x\text{, which is}\gt0$. $\tan\theta\gt0\text{, and }\tan\theta=\sqrt{x^2-1}$,
$\int\frac{x}{|x|\sqrt{x^2-1}}dx\,=\,\int\frac{\sec\theta}{\sec\theta}\frac{\tan\theta}{\tan\theta}\sec\theta\,d\theta\\\qquad\qquad\qquad=\,\int\sec\theta\,d\theta\\\qquad\qquad\qquad=\,\ln|\sec\theta+\tan\theta|+c\\\qquad\qquad\qquad=\,\ln|x+\sqrt{x^2-1}|+c$.
Then, I = $x\sec^{-1}\,x\;-\;\ln|x+\sqrt{x^2-1}|+c$.
I know that this is the right answer, but as I continue, I get a different answer for $\theta\in(\frac{\pi}2, \pi)$.
When $\theta\in(\frac{\pi}2, \pi)$, $\sec\theta=x\text{, which is }\lt0$, $\tan\theta\lt0\text{, and }\tan\theta=-\sqrt{x^2-1}$,
$\int\frac{x}{|x|\sqrt{x^2-1}}dx\,=\,\int\frac{\sec\theta}{-\sec\theta}\frac{\tan\theta}{-\tan\theta}\sec\theta\,d\theta\\\qquad\qquad\qquad=\,\int\sec\theta\,d\theta\\\qquad\qquad\qquad=\,\ln|\sec\theta+\tan\theta|+c\\\qquad\qquad\qquad=\,\ln|x-\sqrt{x^2-1}|+c\\\qquad\qquad\qquad=\,-\ln|x+\sqrt{x^2-1}|+c$
Then, I = $x\sec^{-1}\,x\;\boldsymbol{+}\;\ln|x+\sqrt{x^2-1}|+c$
The answer is $x\sec^{-1}\,x\;\boldsymbol{-}\;\ln|x+\sqrt{x^2-1}|+c\\$, what did I do wrong in the last part?
The anti-derivative valid over all domain $|x|\ge 1$ is obtained as follows \begin{align} \int \sec^{-1}x \ dx =&\ x\sec^{-1}x - \int \frac x{|x|\sqrt{x^2-1}}dx\\ =&\ x\sec^{-1}x - \int \frac 1{\sqrt{|x|^2-1}}d(|x|)\\ =&\ x\sec^{-1}x - \cosh^{-1}(|x|)\ =x\sec^{-1}x - \ln\left(|x|+\sqrt{x^2-1}\right) \end{align} Your or the cited answer is only valid for a sub-domain.