We need to evaluate $$I = \int x\ln\left(x^2 + a^2 + \sqrt{x^2 - a^2}\right)\, \mathrm{d}x$$ so we using $u = \ln f(x)$ and $\mathrm{d}v = x$ so that $\mathrm{d}u = \frac{f'(x)}{f(x)}$ and $v = \frac{x^2}{2}$ (where $f(x) =$ argument of the logarithm yielding:
$$I = \frac{x^2}{2}\ln f(x) - \frac{1}{2} \int \frac{x^2\left(2x + x(x^2 - a^2)^{-1/2}\right)}{x^2 + a^2 + \sqrt{x^2 - a^2}} \, \mathrm{d}x$$
I can't seem to progress from there, I was wondering if you could provide me with any pushes/solutions in the right direction or an altogether new way of approaching the problem cleverly. Thank you!
Using substitution will be more successful that 'by parts'.
Let $t=x^2-a^2$ so $dt=2xdx$
Then the integral becomes: $$ I=\frac{1}{2}\int \ln(t + \sqrt{t}+2a^2)dt$$
Next using 'by parts':
Let $u=\ln(t+\sqrt{t}+2a^2)$ and let $dv=1$
We get: $v=t$ and $$du = \frac{1+\frac{1}{2}t^{-\frac{1}{2}}}{t+\sqrt{t}+2a^2}dt$$
$$I=t \ln(t+\sqrt{t}+2a^2)-\int\frac{1+\frac{1}{2}t^{-\frac{1}{2}}}{t+\sqrt{t}+2a^2} tdt$$ $$I=t \ln(t+\sqrt{t}+2a^2)-\int\frac{t+\frac{1}{2}\sqrt{t}}{t+\sqrt{t}+2a^2} dt$$ Next let: $t=w^2$ so $dt=2wdw$ $$I=t \ln(t+\sqrt{t}+2a^2)-\int\frac{w^2+\frac{1}{2}w}{w^2+w+2a^2} 2wdw$$ $$I=t \ln(t+\sqrt{t}+2a^2)-\int\frac{2w^3+w^2}{w^2+w+2a^2} dw$$
$$I=t \ln(t+\sqrt{t}+2a^2)-\int (2w-1) dw-\int\frac{w-2a^2w+a^2}{w^2+w+2a^2} dw$$ $$I=t \ln(t+\sqrt{t}+2a^2)-w^2+w-\int\frac{\frac{1}{2}(2w+1)(1-2a^2)-\frac{1}{2}+2a^2}{w^2+w+2a^2} dw$$ $$I=t \ln(t+\sqrt{t}+2a^2)-w^2+w-\int\frac{\frac{1}{2}(2w+1)(1-2a^2)}{w^2+w+2a^2} dw-\int\frac{-\frac{1}{2}+2a^2}{w^2+w+2a^2} dw$$ For the first integral (on the last line) let: $y=w^2+w+a^2$ so $dy=(2w+1)dw$ $$I=t \ln(t+\sqrt{t}+2a^2)-w^2+w-\int\frac{\frac{1}{2}(1-2a^2)}{y} dy-\int\frac{-\frac{1}{2}+2a^2}{(w+\frac{1}{2})^2+2a^2-\frac{1}{4}} dw$$ $$I=t \ln(t+\sqrt{t}+2a^2)-w^2+w-\frac{1}{2}(1-2a^2)\ln|y|-\frac{-\frac{1}{2}+2a^2}{\sqrt{2a^2-\frac{1}{4}}}\arctan\left(\frac{w+\frac{1}{2}}{\sqrt{2a^2-\frac{1}{4}}}\right)+c,c\in\mathbb{R}$$
Then substitute back in the various expressions of $x$.
**Comparing this to wolframalpha's result I can tell I've messed up my constants along the way but the main points - which integration techniques to use are there.
P.S. I'll see if I can find time to improve this answer but I've already spend ages on it.
P.P.S. This would be significantly less messy looking and easier to do if $x^2+a^2$ and $x^2-a^2$ were the same. Did you copy the question down correctly??