Define the function $\mathcal{I}:\mathbb{R}^{2}\rightarrow\mathbb{R}$ by the definite integral
$$\mathcal{I}{\left(a,b\right)}:=\int_{0}^{1}\mathrm{d}x\,\frac{\operatorname{arsinh}{\left(ax\right)}\operatorname{arsinh}{\left(bx\right)}}{x},$$
where the inverse hyperbolic sine is given by
$$\operatorname{arsinh}{\left(z\right)}:=\int_{0}^{z}\mathrm{d}t\,\frac{1}{\sqrt{1+t^{2}}}=\ln{\left(z+\sqrt{1+z^{2}}\right)};~~~\small{z\in\mathbb{R}}.$$
Basic properties of $\mathcal{I}$ include:
$$\mathcal{I}{\left(a,b\right)}=\mathcal{I}{\left(b,a\right)},$$
$$\mathcal{I}{\left(0,b\right)}=\mathcal{I}{\left(a,0\right)}=0,$$
$$\mathcal{I}{\left(-a,b\right)}=\mathcal{I}{\left(a,-b\right)}=-\mathcal{I}{\left(a,b\right)}.$$
It suffices then to consider the $0<a\le b$ case to complete a general evaluation of $\mathcal{I}{\left(a,b\right)}$. The case of equal parameters can be evaluated in terms of polylogarithms after making the appropriate Euler substitution, as we'll show below. Unfortunately, that method doesn't seem to extend to the case of unequal parameters.
Question: Given $0<a<b$, can we find a closed form expression for $\mathcal{I}{\left(a,b\right)}$ in terms of polylogarithms?
Equal Parameter Case: Suppose $a\in\mathbb{R}\land a>0$, and set $\alpha:=\operatorname{arsinh}{\left(a\right)}>0$. We find
$$\begin{align} \mathcal{I}{\left(a,a\right)} &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}\\ &=\int_{0}^{a}\mathrm{d}y\,\frac{\left[\operatorname{arsinh}{\left(y\right)}\right]^{2}}{y};~~~\small{\left[x=a^{-1}y\right]}\\ &=\int_{0}^{\operatorname{arsinh}{\left(a\right)}}\mathrm{d}\tau\,\frac{\tau^{2}\cosh{\left(\tau\right)}}{\sinh{\left(\tau\right)}};~~~\small{\left[y=\sinh{\left(\tau\right)}\right]}\\ &=\int_{0}^{\alpha}\mathrm{d}\tau\,\tau^{2}\coth{\left(\tau\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}-\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\ln{\left(\sinh{\left(\tau\right)}\right)};~~~\small{I.B.P.}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}-\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\ln{\left(\frac{e^{\tau}-e^{-\tau}}{2}\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\ln{\left(\frac{2e^{-\tau}}{1-e^{-2\tau}}\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\left[\ln{\left(2\right)}-\tau-\ln{\left(1-e^{-2\tau}\right)}\right]\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\ln{\left(2\right)}-\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau^{2}-\int_{0}^{\alpha}\mathrm{d}\tau\,2\tau\ln{\left(1-e^{-2\tau}\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\alpha^{2}\ln{\left(2\right)}-\frac23\alpha^{3}-\int_{0}^{\alpha}\mathrm{d}\tau\,\tau\frac{d}{d\tau}\operatorname{Li}_{2}{\left(e^{-2\tau}\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\alpha^{2}\ln{\left(2\right)}-\frac23\alpha^{3}-\alpha\operatorname{Li}_{2}{\left(e^{-2\alpha}\right)}+\int_{0}^{\alpha}\mathrm{d}\tau\,\operatorname{Li}_{2}{\left(e^{-2\tau}\right)}\\ &=\alpha^{2}\ln{\left(\sinh{\left(\alpha\right)}\right)}+\alpha^{2}\ln{\left(2\right)}-\frac23\alpha^{3}-\alpha\operatorname{Li}_{2}{\left(e^{-2\alpha}\right)}-\frac12\operatorname{Li}_{3}{\left(e^{-2\alpha}\right)}+\frac12\operatorname{Li}_{3}{\left(1\right)}\\ &=\frac12\zeta{\left(3\right)}-\frac12\operatorname{Li}_{3}{\left(e^{-2\alpha}\right)}-\alpha\operatorname{Li}_{2}{\left(e^{-2\alpha}\right)}+\alpha^{2}\ln{\left(2\sinh{\left(\alpha\right)}\right)}-\frac23\alpha^{3}.\\ \end{align}$$
Recall the addition formula for the inverse hyperbolic sine:
$$\operatorname{arsinh}{\left(z\right)}\pm\operatorname{arsinh}{\left(w\right)}=\operatorname{arsinh}{\left(z\sqrt{1+w^{2}}\pm w\sqrt{1+z^{2}}\right)};~~~\small{z\in\mathbb{R}\land w\in\mathbb{R}}.$$
For $(a,b)\in\mathbb{R}^{2}\land0<a<b$, we have
$$\begin{align} 2\mathcal{I}{\left(a,b\right)} &=2\int_{0}^{1}\mathrm{d}x\,\frac{\operatorname{arsinh}{\left(ax\right)}\operatorname{arsinh}{\left(bx\right)}}{x}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(ax\right)}\right]^{2}+\left[\operatorname{arsinh}{\left(bx\right)}\right]^{2}-\left[\operatorname{arsinh}{\left(bx\right)}-\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}+\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\right)}\right]^{2}}{x}-\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\right)}-\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\sqrt{1+a^{2}x^{2}}-ax\sqrt{1+b^{2}x^{2}}\right)}\right]^{2}}{x}.\\ \end{align}$$
The last line above suggests the quite complicated substitution
$$bx\sqrt{1+a^{2}x^{2}}-ax\sqrt{1+b^{2}x^{2}}=y.$$
It can be shown that any $(x,y)$ satisfying the substitution relation also satisfies the polynomial
$$\left[\left(b^{2}-a^{2}\right)^{2}-4a^{2}b^{2}y^{2}\right]x^{4}-2\left(a^{2}+b^{2}\right)y^{2}x^{2}+y^{4}=0,$$
which can be solved for $x^{2}$ using the quadratic formula, yielding
$$x^{2}=\frac{y^{2}}{a^{2}+b^{2}\mp2ab\sqrt{1+y^{2}}},$$
$$\implies x=\frac{y}{\sqrt{a^{2}+b^{2}-2ab\sqrt{1+y^{2}}}},$$
where the sign ambiguities have been chosen to give the correct limiting behavior as $x\to\infty$.
Differentiating both sides by $y$, we have
$$\frac{d}{dy}x^{2}=\frac{d}{dy}\left(\frac{y^{2}}{a^{2}+b^{2}-2ab\sqrt{1+y^{2}}}\right),$$
$$\implies x\frac{dx}{dy}=\frac{y}{a^{2}+b^{2}-2ab\sqrt{1+y^{2}}}+\frac{aby^{3}}{\left(a^{2}+b^{2}-2ab\sqrt{1+y^{2}}\right)^{2}\sqrt{1+y^{2}}},$$
$$\implies\frac{1}{x}\frac{dx}{dy}=\frac{1}{y}+\frac{aby}{\left(a^{2}+b^{2}-2ab\sqrt{1+y^{2}}\right)\sqrt{1+y^{2}}}.$$
Set $\eta:=\operatorname{arsinh}{\left(b\sqrt{1+a^{2}}-a\sqrt{1+b^{2}}\right)}>0$. Then, $\sinh{\left(\eta\right)}=b\sqrt{1+a^{2}}-a\sqrt{1+b^{2}}$. Continuing from where we left off in the evaluation of $\mathcal{I}$, we find the following upon applying the substitution:
$$\begin{align} 2\mathcal{I}{\left(a,b\right)} &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\sqrt{1+a^{2}x^{2}}-ax\sqrt{1+b^{2}x^{2}}\right)}\right]^{2}}{x}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\int_{0}^{b\sqrt{1+a^{2}}-a\sqrt{1+b^{2}}}\mathrm{d}y\,\left[\frac{1}{y}+\frac{aby}{\left(a^{2}+b^{2}-2ab\sqrt{1+y^{2}}\right)\sqrt{1+y^{2}}}\right]\\ &~~~~~\times\left[\operatorname{arsinh}{\left(y\right)}\right]^{2};~~~\small{\left[bx\sqrt{1+a^{2}x^{2}}-ax\sqrt{1+b^{2}x^{2}}=y\right]}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}\\ &~~~~~-\int_{0}^{\sinh{\left(\eta\right)}}\mathrm{d}y\,\left[\frac{1}{y}+\frac{aby}{\left(a^{2}+b^{2}-2ab\sqrt{1+y^{2}}\right)\sqrt{1+y^{2}}}\right]\left[\operatorname{arsinh}{\left(y\right)}\right]^{2}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\int_{0}^{\sinh{\left(\eta\right)}}\mathrm{d}y\,\frac{\left[\operatorname{arsinh}{\left(y\right)}\right]^{2}}{y}\\ &~~~~~-\int_{0}^{\sinh{\left(\eta\right)}}\mathrm{d}y\,\frac{aby\left[\operatorname{arsinh}{\left(y\right)}\right]^{2}}{\left(a^{2}+b^{2}-2ab\sqrt{1+y^{2}}\right)\sqrt{1+y^{2}}}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}\\ &~~~~~-\int_{0}^{\eta}\mathrm{d}\tau\,\frac{ab\tau^{2}\sinh{\left(\tau\right)}}{a^{2}+b^{2}-2ab\cosh{\left(\tau\right)}};~~~\small{\left[y=\sinh{\left(\tau\right)}\right]}.\\ \end{align}$$
Setting $p:=\frac{a}{b}$ and $q:=\exp{\left(\eta\right)}=\frac{b+\sqrt{1+b^{2}}}{a+\sqrt{1+a^{2}}}$, we have $0<p<1\land1<q<\frac{1}{p}$, and then
$$\begin{align} 2\mathcal{I}{\left(a,b\right)} &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\int_{0}^{\eta}\mathrm{d}\tau\,\frac{ab\tau^{2}\sinh{\left(\tau\right)}}{a^{2}+b^{2}-2ab\cosh{\left(\tau\right)}}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\int_{0}^{\eta}\mathrm{d}\tau\,\frac{p\tau^{2}\sinh{\left(\tau\right)}}{p^{2}+1-2p\cosh{\left(\tau\right)}}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}\\ &~~~~~-\int_{1}^{\exp{\left(\eta\right)}}\mathrm{d}t\,\frac{1}{t}\cdot\frac{p\left(\frac{t^{2}-1}{2t}\right)\ln^{2}{\left(t\right)}}{p^{2}+1-2p\left(\frac{t^{2}+1}{2t}\right)};~~~\small{\left[\tau=\ln{\left(t\right)}\right]}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\frac12\int_{1}^{\exp{\left(\eta\right)}}\mathrm{d}t\,\frac{1}{t}\cdot\frac{p\left(t^{2}-1\right)\ln^{2}{\left(t\right)}}{\left(p^{2}+1\right)t-p\left(t^{2}+1\right)}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\frac12\int_{1}^{q}\mathrm{d}t\,\frac{p\left(t^{2}-1\right)\ln^{2}{\left(t\right)}}{t\left(1-pt\right)\left(t-p\right)}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\frac12\int_{1}^{q}\mathrm{d}t\,\left[\frac{p}{\left(1-pt\right)}-\frac{p}{t\left(t-p\right)}\right]\ln^{2}{\left(t\right)}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\frac12\int_{1}^{q}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{1-pt}+\frac12\int_{1}^{q}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{t\left(t-p\right)}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\frac12\int_{1}^{q}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{1-pt}\\ &~~~~~+\frac12\int_{\frac{1}{q}}^{1}\mathrm{d}u\,\frac{p\ln^{2}{\left(u\right)}}{1-pu};~~~\small{\left[t=\frac{1}{u}\right]}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\int_{1}^{q}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{2\left(1-pt\right)}+\int_{\frac{1}{q}}^{1}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{2\left(1-pt\right)}.\\ \end{align}$$
The following derivative is readily verified:
$$\frac{d}{dt}\left[\operatorname{Li}_{3}{\left(pt\right)}-\ln{\left(t\right)}\operatorname{Li}_{2}{\left(pt\right)}-\frac12\ln^{2}{\left(t\right)}\ln{\left(1-pt\right)}\right]=\frac{p\ln^{2}{\left(t\right)}}{2\left(1-pt\right)};~~~\small{0<p\land0<t<\frac{1}{p}}.$$
Thus,
$$\begin{align} 2\mathcal{I}{\left(a,b\right)} &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}-\int_{1}^{q}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{2\left(1-pt\right)}+\int_{\frac{1}{q}}^{1}\mathrm{d}t\,\frac{p\ln^{2}{\left(t\right)}}{2\left(1-pt\right)}\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}\\ &~~~~~-\int_{1}^{q}\mathrm{d}t\,\frac{d}{dt}\left[\operatorname{Li}_{3}{\left(pt\right)}-\ln{\left(t\right)}\operatorname{Li}_{2}{\left(pt\right)}-\frac12\ln^{2}{\left(t\right)}\ln{\left(1-pt\right)}\right]\\ &~~~~~+\int_{\frac{1}{q}}^{1}\mathrm{d}t\,\frac{d}{dt}\left[\operatorname{Li}_{3}{\left(pt\right)}-\ln{\left(t\right)}\operatorname{Li}_{2}{\left(pt\right)}-\frac12\ln^{2}{\left(t\right)}\ln{\left(1-pt\right)}\right]\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}\\ &~~~~~-\left[\operatorname{Li}_{3}{\left(pq\right)}-\ln{\left(q\right)}\operatorname{Li}_{2}{\left(pq\right)}-\frac12\ln^{2}{\left(q\right)}\ln{\left(1-pq\right)}\right]+\operatorname{Li}_{3}{\left(p\right)}\\ &~~~~~+\operatorname{Li}_{3}{\left(p\right)}-\left[\operatorname{Li}_{3}{\left(\frac{p}{q}\right)}-\ln{\left(\frac{1}{q}\right)}\operatorname{Li}_{2}{\left(\frac{p}{q}\right)}-\frac12\ln^{2}{\left(\frac{1}{q}\right)}\ln{\left(1-\frac{p}{q}\right)}\right]\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}+2\operatorname{Li}_{3}{\left(p\right)}-\operatorname{Li}_{3}{\left(pq\right)}-\operatorname{Li}_{3}{\left(\frac{p}{q}\right)}\\ &~~~~~+\ln{\left(q\right)}\operatorname{Li}_{2}{\left(pq\right)}-\ln{\left(q\right)}\operatorname{Li}_{2}{\left(\frac{p}{q}\right)}+\frac12\ln^{2}{\left(q\right)}\left[\ln{\left(1-pq\right)}+\ln{\left(1-\frac{p}{q}\right)}\right]\\ &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\eta\right)},\sinh{\left(\eta\right)}\right)}+2\operatorname{Li}_{3}{\left(\frac{a}{b}\right)}-\operatorname{Li}_{3}{\left(\frac{a}{b}e^{\eta}\right)}-\operatorname{Li}_{3}{\left(\frac{a}{b}e^{-\eta}\right)}\\ &~~~~~+\eta\operatorname{Li}_{2}{\left(\frac{a}{b}e^{\eta}\right)}-\eta\operatorname{Li}_{2}{\left(\frac{a}{b}e^{-\eta}\right)}+\frac12\eta^{2}\ln{\left(1+\frac{a^{2}}{b^{2}}-\frac{2a}{b}\cosh{\left(\eta\right)}\right)}.\blacksquare\\ \end{align}$$
Since the equal parameter case for $\mathcal{I}$ has already been solved, we are done!