$$I=\int \frac{\ln(t+\sqrt{t^2+1)}}{1+t^2} \, dt$$
i used substitution $t=\tan y$ so
$$I=\int \ln(\sec y+\tan y)dy$$
Using integration by parts we get:
$$I=y \ln(\sec y+\tan y)-\int y \sec ydy$$
any clue here?
I also tried in other way:
Knowing that $$\int \frac{\ln(t+\sqrt{t^2+1)}}{\sqrt{1+t^2}} \, dt=\frac{1}{2}\left(\ln(t+\sqrt{t^2+1)}\right)^2$$ $$I=\int \frac{1}{\sqrt{t^2+1}} \times \frac{\ln(t+\sqrt{t^2+1)}}{\sqrt{1+t^2}} \, dt$$
Now using integration by parts we get:
$$I=\frac{1}{\sqrt{t^2+1}}\frac{1}{2}\left(\ln(t+\sqrt{t^2+1)}\right)^2+\frac{1}{2}\int \frac{tdt}{(t^2+1)^{\frac{3}{2}}}\left(\ln(t+\sqrt{t^2+1)}\right)^2$$
Consider the following trigonometric equality $$\begin{align} t+\sqrt{t^2+1} &= u\\ t+\sqrt{t^2+1}&= e^{\sinh^{-1}t}\\ \ln(t+\sqrt{t^2+1})&=\ln(e^{\sinh^{-1}t})\\ \ln(t+\sqrt{x^2+1})&=\sinh^{-1}t \\ \end{align}$$ $$\ln(t+\sqrt{x^2+1})= \sinh^{-1}(t) = \operatorname{arsinh}(t) $$ In this way we can rewrite the integral as $$\begin{align} I = \int \frac{\ln(t+\sqrt{t^2+1)}}{1+t^2} \, dt &= \int \frac{\operatorname{arsinh}(t)}{(t - i)(t+i)} \, dt \\ &=\int \left(\frac{i\operatorname{arsinh}(t)}{2(t + i)} - \frac{i\operatorname{arsinh}(t)}{2(t-i)}\right) \, dt\\ &= \frac{i}{2}\underbrace{\int \frac{\operatorname{arsinh}(t)}{t + i}\,dt}_{I_1} - \frac{i}{2}\underbrace{\int \frac{\operatorname{arsinh}(t)}{t-i} \, dt}_{I_2} \end{align}$$ Now we solve $I_1$:
$$\begin{align} I_1 &= \int \frac{\operatorname{arsinh}(t)}{t + i}\,dt &&\small{\text{Substitute: } u=t+i \to dt=du}\\ &=\int \frac{\operatorname{arsinh}(u -i)}{u}\,du && \small{\text{Substitute: }v=\operatorname{arsinh}(u-i) \to du=\sqrt{(u-i)^2+1}\,dv}\\[0,1pt] &&&\small{\text{Let: }\operatorname{sinh}^2(u)=\operatorname{cosh}^2(u)-1\\ \qquad u=\operatorname{sinh}(v)+i \implies \frac{1}{u}=\frac{1}{\operatorname{sinh}(v)+i}}\\ &=\int \frac{v\operatorname{cosh}(v)}{\operatorname{sinh}(v)+i}\,dv &&\small{\text{Write as hyperbolic functions}}\\ &=\int \frac{v(e^v+e^{-v})}{2\left(\frac{e^v-e^{-v}}{2}+i\right)}\,dv =\int \frac{v(e^v+e^{-v})}{e^v-e^{-v}+2i}\,dv =\int\frac{v(e^v-i)}{e^v+i}\,dv &&\small{\text{Substitute: } w=e^v+i \to dv=e^{-v}\,dw}\\[0,1pt] &&&\small{\text{Let: }v=\ln(w-i)}\\ &= \int \frac{(w-2i)\ln(w-i)}{w(w-i)}\,dw = \int\left(\frac{2\ln(w-i)}{w}-\frac{\ln(w-i)}{w-i}\right)\,dw\\ &=2\underbrace{\int\frac{\ln(w-i)}{w}\,dw}_{И_1}-\underbrace{\int\frac{\ln(w-i)}{w-i}\,dw}_{И_2} \end{align}$$
$$\begin{align} И_1 &= 2\int\frac{\ln(w-i)}{w}\,dw&\\ &= 2\int\left(\frac{\ln(iw+1)}{w}+\frac{\ln(-i)}{w}\right)\,dw&\\ &= 2\int\frac{\ln(iw+1)}{w}+2\ln(-i)\int\frac{1}{w}\,dw&\\ &= 2\int\frac{\ln(iw+1)}{w}+2\ln(-i)\ln(w)&\small{\text{Substitute: } y=-iw \to dw=i\,dy}\\ &&\small{\int\frac{\ln(iw+1)}{w} = -\int-\frac{\ln(1-y)}{y}\,dy -\operatorname{Li_2}(y) = -\operatorname{Li_2}(-iw)}\\ &= 2\ln(-i)\ln(w) -2\operatorname{Li_2}(-iw) \\\\\\ И_2 &= \int\frac{\ln(w-i)}{w-i}\,dw&\small{\text{Substitute: } y=\ln(w−i) \to dw=(w−i)\,dy}\\ &=\int y\,dy = \frac{y^2}{2} = \frac{\ln^2(w-i)}{2} \end{align}$$
So $$\begin{align} I_1 &= 2\ln(-i)\ln(w) -2\operatorname{Li_2}(-iw) - \frac{\ln^2(w-i)}{2}\\ &= 2\ln(-i)\ln(e^v+i) -2\operatorname{Li_2}\left(-i\left(e^v+i\right)\right) - \frac{\ln^2\left(\left(e^v+i\right)-i\right)}{2}\\ &= 2\ln(-i)\ln(e^v+i) -2\operatorname{Li_2}\left(-i\left(e^v+i\right)\right) - \frac{v^2}{2}\\ \small&= 2\ln(-i)\ln(e^{\operatorname{arsinh}(u-i)}+i) -2\operatorname{Li_2}\left(-i\left(e^{\operatorname{arsinh}(u-i)}+i\right)\right) - \frac{(\operatorname{arsinh}(u-i))^2}{2}\\ &= 2\ln(-i)\ln(e^{\operatorname{arsinh}(t)}+i) -2\operatorname{Li_2}\left(-i\left(e^{\operatorname{arsinh}(t)}+i\right)\right) - \frac{\operatorname{arsinh}^2(t)}{2}\\ \end{align}$$
And now we solve in the same way the integral $I_2$: $$\begin{align} I_2 &= \int \frac{\operatorname{arsinh}(t)}{t-i} \, dt&&\small{\text{Substitute: } u=t-i \to dt=du}\\ &=\int \frac{\operatorname{arsinh}(u +i)}{u}\,du && \\ &=\vdots\\ &=2\underbrace{\int\frac{\ln(w+i)}{w}\,dw}_{И_1}-\underbrace{\int\frac{\ln(w+i)}{w+i}\,dw}_{И_2}\\ &= 2\ln(i)\ln(w) -2\operatorname{Li_2}(iw) - \frac{\ln^2(w+i)}{2}\\ &= 2\ln(i)\ln(e^{\operatorname{arsinh}(t)}-i) -2\operatorname{Li_2}\left(i\left(e^{\operatorname{arsinh}(t)}-i\right)\right) - \frac{\operatorname{arsinh}^2(t)}{2}\\ \end{align}$$
Therefore $$ I = \frac{i}{2}I_1 - \frac{i}{2}I_2$$ $$\tiny{\frac{i}{2}\left(2\ln(-i)\ln(e^{\operatorname{arsinh}(t)}+i) -2\operatorname{Li_2}\left(-i\left(e^{\operatorname{arsinh}(t)}+i\right)\right) - \frac{\operatorname{arsinh}^2(t)}{2}\right) - \frac{i}{2}\left(2\ln(i)\ln(e^{\operatorname{arsinh}(t)}-i) -2\operatorname{Li_2}\left(i\left(e^{\operatorname{arsinh}(t)}-i\right)\right) - \frac{\operatorname{arsinh}^2(t)}{2}\right)}$$
$$\small{ i\ln(-i)\ln(e^{\operatorname{arsinh}(t)}+i) -i\operatorname{Li_2}\left(-i\left(e^{\operatorname{arsinh}(t)}+i\right)\right) - i\ln(i)\ln(e^{\operatorname{arsinh}(t)}-i) +i\operatorname{Li_2}\left(i\left(e^{\operatorname{arsinh}(t)}-i\right)\right)} $$
See Polylogarithm