Define the function $\mathcal{I}:\mathbb{R}_{>0}^{2}\rightarrow\mathbb{R}$ via the definite integral
$$\mathcal{I}{\left(a,b\right)}:=\int_{0}^{1}\mathrm{d}x\,\frac{\operatorname{arsinh}{\left(ax\right)}\operatorname{arsinh}{\left(bx\right)}}{x},\tag{1}$$
where for our purposes here the inverse hyperbolic sine may be given by the logarithmic expression
$$\operatorname{arsinh}{\left(z\right)}:=\ln{\left(z+\sqrt{z^{2}+1}\right)};~~~\small{z\in\mathbb{R}}.\tag{2}$$
Obviously, $\mathcal{I}{\left(a,b\right)}$ is symmetric under interchanging $a$ and $b$, so it will suffice to consider the $0<a\le b$ case.
For the special case of equal parameters, it's not too difficult to show (either by differentiation or direct methods) that
$$\mathcal{I}{\left(a,a\right)}=\frac12S_{1,2}{\left(\frac{2a}{a+\sqrt{a^{2}+1}}\right)}+\frac13\left[\operatorname{arsinh}{\left(a\right)}\right]^{3};~~~\small{a\in\mathbb{R}_{>0}},\tag{3}$$
where the Nielsen generalized polylogarithm $S_{1,2}$ is typically defined by the integral representation
$$S_{1,2}{\left(z\right)}:=\int_{0}^{1}\mathrm{d}t\,\frac{\ln^{2}{\left(1-zt\right)}}{2t}=\int_{0}^{z}\mathrm{d}x\,\frac{\ln^{2}{\left(1-x\right)}}{2x};~~~\small{z\in\left(-\infty,1\right]}.\tag{4}$$
Problem: Given $\left(a,b\right)\in\mathbb{R}_{>0}^{2}$ such that $a<b$, can we obtain a closed-form expression for $\mathcal{I}{\left(a,b\right)}$ in terms of polylogarithms and elementary functions?.
Define the auxiliary function $\mathcal{J}$ in terms of $\mathcal{I}$ as follows:
$$\mathcal{J}{\left(a,b\right)}:=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-2\mathcal{I}{\left(a,b\right)};~~~\small{\left(a,b\right)\in\mathbb{R}_{>0}^{2}}.$$
Recall the addition formula for the inverse hyperbolic sine:
$$\operatorname{arsinh}{\left(z\right)}\pm\operatorname{arsinh}{\left(w\right)}=\operatorname{arsinh}{\left(z\sqrt{w^{2}+1}\pm w\sqrt{z^{2}+1}\right)};~~~\small{\left(z,w\right)\in\mathbb{R}^{2}}.$$
Supposing $0<a<b$, we then have
$$\begin{align} \mathcal{J}{\left(a,b\right)} &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-2\mathcal{I}{\left(a,b\right)}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}+\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\right)}\right]^{2}}{x}\\ &~~~~~-2\int_{0}^{1}\mathrm{d}x\,\frac{\operatorname{arsinh}{\left(ax\right)}\operatorname{arsinh}{\left(bx\right)}}{x}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(ax\right)}\right]^{2}+\left[\operatorname{arsinh}{\left(bx\right)}\right]^{2}-2\operatorname{arsinh}{\left(ax\right)}\operatorname{arsinh}{\left(bx\right)}}{x}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\right)}-\operatorname{arsinh}{\left(ax\right)}\right]^{2}}{x}\\ &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\sqrt{a^{2}x^{2}+1}-ax\sqrt{b^{2}x^{2}+1}\right)}\right]^{2}}{x}.\\ \end{align}$$
Consider the following substitution:
$$bx\sqrt{a^{2}x^{2}+1}-ax\sqrt{b^{2}x^{2}+1}=u.$$
Eliminating the radicals from the substitution relation yields the following quadratic equation in $x^{2}$:
$$\left[\left(a^{2}+b^{2}\right)^{2}-4a^{2}b^{2}\left(1+u^{2}\right)\right]x^{4}-2\left(a^{2}+b^{2}\right)u^{2}x^{2}+u^{4}=0.$$
Solving for $x^{2}$, we obtain
$$x^{2}=\frac{u^{2}}{a^{2}+b^{2}+2ab\sqrt{1+u^{2}}},$$
$$\implies \frac{1}{x}\frac{dx}{du}=\frac{1}{u}-\frac{abu}{\left(a^{2}+b^{2}+2ab\sqrt{1+u^{2}}\right)\sqrt{1+u^{2}}}.$$
Noting that $b\sqrt{a^{2}+1}-a\sqrt{b^{2}+1}>0$, set $\gamma:=\operatorname{arsinh}{\left(b\sqrt{a^{2}+1}-a\sqrt{b^{2}+1}\right)}$. Then,
$$\begin{align} \mathcal{J}{\left(a,b\right)} &=\int_{0}^{1}\mathrm{d}x\,\frac{\left[\operatorname{arsinh}{\left(bx\sqrt{a^{2}x^{2}+1}-ax\sqrt{b^{2}x^{2}+1}\right)}\right]^{2}}{x}\\ &=\int_{0}^{b\sqrt{a^{2}+1}-a\sqrt{b^{2}+1}}\mathrm{d}u\,\left[\frac{1}{u}-\frac{abu}{\left(a^{2}+b^{2}+2ab\sqrt{1+u^{2}}\right)\sqrt{1+u^{2}}}\right]\left[\operatorname{arsinh}{\left(u\right)}\right]^{2}\\ &=\int_{0}^{\sinh{\left(\gamma\right)}}\mathrm{d}u\,\left[\frac{1}{u}-\frac{abu}{\left(a^{2}+b^{2}+2ab\sqrt{1+u^{2}}\right)\sqrt{1+u^{2}}}\right]\left[\operatorname{arsinh}{\left(u\right)}\right]^{2}\\ &=\int_{0}^{\sinh{\left(\gamma\right)}}\mathrm{d}u\,\frac{\left[\operatorname{arsinh}{\left(u\right)}\right]^{2}}{u}-\int_{0}^{\sinh{\left(\gamma\right)}}\mathrm{d}u\,\frac{abu\left[\operatorname{arsinh}{\left(u\right)}\right]^{2}}{\left(a^{2}+b^{2}+2ab\sqrt{1+u^{2}}\right)\sqrt{1+u^{2}}}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\int_{0}^{\sinh{\left(\gamma\right)}}\mathrm{d}u\,\frac{abu\left[\operatorname{arsinh}{\left(u\right)}\right]^{2}}{\left(a^{2}+b^{2}+2ab\sqrt{1+u^{2}}\right)\sqrt{1+u^{2}}}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\int_{0}^{\gamma}\mathrm{d}x\,\frac{abx^{2}\sinh{\left(x\right)}}{a^{2}+b^{2}+2ab\cosh{\left(x\right)}};~~~\small{\left[\operatorname{arsinh}{\left(u\right)}=x\right]}.\\ \end{align}$$
Next, noting that $0<\frac{2ab}{a^{2}+b^{2}}<1$, set $\theta:=\arcsin{\left(\frac{2ab}{a^{2}+b^{2}}\right)}$. Then,
$$\begin{align} \mathcal{J}{\left(a,b\right)} &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\int_{0}^{\gamma}\mathrm{d}x\,\frac{abx^{2}\sinh{\left(x\right)}}{a^{2}+b^{2}+2ab\cosh{\left(x\right)}}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\int_{0}^{\gamma}\mathrm{d}x\,\frac{x^{2}\sin{\left(\theta\right)}\sinh{\left(x\right)}}{1+\sin{\left(\theta\right)}\cosh{\left(x\right)}}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~+\int_{0}^{\gamma}\mathrm{d}x\,x\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(x\right)}\right)};~~~\small{I.B.P.}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{1}{y}\ln{\left(y\right)}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\ln{\left(y\right)}\right)}\right)};~~~\small{\left[x=\ln{\left(y\right)}\right]}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{1}{y}\ln{\left(y\right)}\ln{\left(1+\frac{y^{2}+1}{2y}\sin{\left(\theta\right)}\right)}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{1}{y}\ln{\left(y\right)}\ln{\left(\frac{y^{2}+1+2y\csc{\left(\theta\right)}}{2y\csc{\left(\theta\right)}}\right)}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{1}{y}\ln{\left(y\right)}\left[-\ln{\left(2\csc{\left(\theta\right)}\right)}-\ln{\left(y\right)}+\ln{\left(y^{2}+2y\csc{\left(\theta\right)}+1\right)}\right]\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(2\csc{\left(\theta\right)}\right)}\ln{\left(y\right)}}{y}-\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln^{2}{\left(y\right)}}{y}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y^{2}+2y\csc{\left(\theta\right)}+1\right)}}{y}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}-\frac13\gamma^{3}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y^{2}+2y\csc{\left(\theta\right)}+1\right)}}{y}.\\ \end{align}$$
Observe that $y^{2}+2y\csc{\left(\theta\right)}+1=\left[y+\tan{\left(\frac{\theta}{2}\right)}\right]\left[y+\cot{\left(\frac{\theta}{2}\right)}\right]$, and $\tan{\left(\frac{\theta}{2}\right)}=\frac{a}{b}\land\cot{\left(\frac{\theta}{2}\right)}=\frac{b}{a}$.
Given $z>1\land p>0$, we can derive the following integration formula:
$$\begin{align} \int_{1}^{z}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y+p\right)}}{y} &=\int_{1}^{z}\mathrm{d}y\,\frac{d}{dy}\left[\operatorname{Li}_{3}{\left(-\frac{y}{p}\right)}-\ln{\left(y\right)}\operatorname{Li}_{2}{\left(-\frac{y}{p}\right)}+\frac12\ln{\left(p\right)}\ln^{2}{\left(y\right)}\right]\\ &=\operatorname{Li}_{3}{\left(-\frac{z}{p}\right)}-\operatorname{Li}_{3}{\left(-\frac{1}{p}\right)}-\ln{\left(z\right)}\operatorname{Li}_{2}{\left(-\frac{z}{p}\right)}+\frac12\ln{\left(p\right)}\ln^{2}{\left(z\right)}.\\ \end{align}$$
Then,
$$\begin{align} \mathcal{J}{\left(a,b\right)} &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}-\frac13\gamma^{3}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y^{2}+2y\csc{\left(\theta\right)}+1\right)}}{y}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}-\frac13\gamma^{3}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y+\tan{\left(\frac{\theta}{2}\right)}\right)}}{y}\\ &~~~~~+\int_{1}^{\exp{\left(\gamma\right)}}\mathrm{d}y\,\frac{\ln{\left(y\right)}\ln{\left(y+\cot{\left(\frac{\theta}{2}\right)}\right)}}{y}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}-\frac13\gamma^{3}\\ &~~~~~+\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}-\operatorname{Li}_{3}{\left(-\frac{1}{\tan{\left(\frac{\theta}{2}\right)}}\right)}-\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}+\frac12\gamma^{2}\ln{\left(\tan{\left(\frac{\theta}{2}\right)}\right)}\\ &~~~~~+\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}-\operatorname{Li}_{3}{\left(-\frac{1}{\cot{\left(\frac{\theta}{2}\right)}}\right)}-\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}+\frac12\gamma^{2}\ln{\left(\cot{\left(\frac{\theta}{2}\right)}\right)}\\ &=\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}-\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}\\ &~~~~~-\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}-\frac13\gamma^{3}\\ &~~~~~+\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}-\operatorname{Li}_{3}{\left(-\frac{1}{\tan{\left(\frac{\theta}{2}\right)}}\right)}-\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}\\ &~~~~~+\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}-\operatorname{Li}_{3}{\left(-\frac{1}{\cot{\left(\frac{\theta}{2}\right)}}\right)}-\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}.\\ \end{align}$$
Thus,
$$\begin{align} 2\mathcal{I}{\left(a,b\right)} &=\mathcal{I}{\left(a,a\right)}+\mathcal{I}{\left(b,b\right)}-\mathcal{I}{\left(\sinh{\left(\gamma\right)},\sinh{\left(\gamma\right)}\right)}\\ &~~~~~+\frac12\gamma^{2}\ln{\left(1+\sin{\left(\theta\right)}\cosh{\left(\gamma\right)}\right)}+\frac12\gamma^{2}\ln{\left(2\csc{\left(\theta\right)}\right)}+\frac13\gamma^{3}\\ &~~~~~-\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}+\operatorname{Li}_{3}{\left(-\frac{1}{\tan{\left(\frac{\theta}{2}\right)}}\right)}+\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\tan{\left(\frac{\theta}{2}\right)}}\right)}\\ &~~~~~-\operatorname{Li}_{3}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}+\operatorname{Li}_{3}{\left(-\frac{1}{\cot{\left(\frac{\theta}{2}\right)}}\right)}+\gamma\operatorname{Li}_{2}{\left(-\frac{\exp{\left(\gamma\right)}}{\cot{\left(\frac{\theta}{2}\right)}}\right)}.\blacksquare\\ \end{align}$$