How to evaluate $$A=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}+1\right)}{\sqrt{1-x^2}} \, dx$$See the details here from a similar question: Evaluating $\int_0^1 \frac{z \log ^2\left(\sqrt{z^2+1}-1\right)}{\sqrt{1-z^2}} \, dz$. By applying the same process, I find: $$A=\int_0^1\ln^2\left(\sqrt{2-y^2}+1\right)dy=\int_1^{\sqrt2}\frac {x\log^2 (x+1)}{\sqrt{2-x^2}}dx=\sqrt2 \int_{0}^{\pi/4}\cos\varphi\ln^2\left(\sqrt2 \cos\varphi+1\right)d\varphi.$$
$$A=\ln^22+32(1+\sqrt2)\int_{0}^{\sqrt2-1}\frac{t^2}{\left(1+t^2\right)^2(t+1+\sqrt2)(-t+1+\sqrt2)}\ln\left(\sqrt2\, \frac{1-t^2}{1+t^2}+1\right)dt.$$ Now,put $$A=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}+1\right)}{\sqrt{1-x^2}} \, dx, B=\int_0^1 \frac{x \log ^2\left(\sqrt{x^2+1}-1\right)}{\sqrt{1-x^2}} \, dx.$$
We have:
$$A-B=4\int_0^{1}\frac {x\log x\log(\sqrt{1+x^2}+1)}{\sqrt{1-x^2}}dx-4\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx$$ $$A+B=-2\int_0^{1}\frac {x\log (\sqrt{1+x^2}-1)\log(\sqrt{1+x^2}+1)}{\sqrt{1-x^2}}dx+4\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx$$ $$\int_0^{1}\frac {x\log^2 x}{\sqrt{1-x^2}}dx=\ln^22-\frac{\pi^2}{12}-2\ln2+2.$$ But how to deduce these integrals.(using Mathematica?)
Substitute $x^2=\sin 2\theta$ to get $\sqrt{1\pm x^2}= \cos\theta\pm\sin\theta$
\begin{align} I=& \int_0^1 \frac{x \log ^2(\sqrt{1+x^2}+1)}{\sqrt{1-x^2}} \, dx\\ =& \ \frac12\int_0^{\pi/2}\ln^2(\cos\theta+\sin\theta+1)(\cos\theta+\sin\theta)\ d\theta\\ =& \int_0^1 \ln^2\frac{2(1+t)}{1+t^2}\ d\left( \frac{t-1}{1+t^2}\right)\>\>\>\>\>\>\>t=\tan\frac{\theta}2\\ \overset{ibp}=&\ \ln^22 + \int_0^1 \ln\frac{2(1+t)}{1+t^2}\left(\frac1{1+t}+\frac{2}{1+t^2}+\frac{4(t^2-1)}{(1+t^2)^2}\right)dt \end{align} where the remaining integrals are readily manageable, given by \begin{align} & \int_0^1 \ln\frac{2(1+t)}{1+t^2}\ \frac1{1+t}\ dt =\frac34\ln^22+\frac{\pi^2}{48}\\ & \int_0^1 \ln\frac{2(1+t)}{1+t^2}\ \frac{1}{1+t^2}\ dt = G-\frac\pi{8}\ln2\\ & \int_0^1 \ln\frac{2(1+t)}{1+t^2}\ \frac{t^2-1}{(1+t^2)^2}\ dt = -\frac\pi8-\frac34\ln2+\frac12 \end{align} Substitute above results into $I$ to obtain $$I=\frac74\ln^22+\frac{\pi^2}{48}+ 2G-\frac\pi4\ln2-\frac\pi2-3\ln2+2 $$