I am trying to prove that $$I=\int_0^1 \frac{\log^2(x)\tanh^{-1}(x)}{1+x^2}dx=\beta(4)-\frac{\pi^2}{12}G$$ where $\beta(s)$ is the Dirichlet Beta function and $G$ is the Catalan's constant. I managed to derive the following series involving polygamma functions but it doesn't seem to be of much help.
$$ \begin{align*} I &=\frac{1}{64}\sum_{n=0}^\infty \frac{\psi_2 \left(\frac{n}{2}+1 \right) -\psi_2\left(\frac{n+1}{2} \right)}{2n+1} \\ &= \frac{1}{8}\sum_{n=1}^\infty \frac{\psi_2(n)}{2n-1}-\frac{1}{32}\sum_{n=1}^\infty\frac{\psi_2\left(\frac{n}{2}\right)}{2n-1} \end{align*} $$
Numerical calculations show that $I \approx 0.235593$.
I was able to solve this problem on my own.
Using integration by parts, $$\begin{align*} &\; \int_0^1 \frac{\log^2(x)\tanh^{-1}(x)}{1+x^2}dx \\ &= -2\int_0^1 \frac{\log(x)\tan^{-1}(x)\tanh^{-1}(x)}{x}dx-\int_0^1 \frac{\log^2(x)\tan^{-1}(x)}{1-x^2}dx \tag{1} \end{align*}$$
I posted the solution to both these integrals on another forum. Here are the links:
http://integralsandseries.prophpbb.com/topic711.html#p3975
http://integralsandseries.prophpbb.com/topic245.html#p1680
$$\begin{align*}\int_0^1\frac{\log(x)\tan^{-1}(x)\tanh^{-1}(x)}{x}dx &= \frac{\pi^2}{16}G-\frac{7\pi\zeta(3)}{32} \tag{2}\\ \int_0^1\frac{\log^2(x)\tan^{-1}(x)}{1-x^2}dx &= -\beta(4)-\frac{\pi^2}{24}G+\frac{7\pi}{16}\zeta(3)\tag{3} \end{align*}$$ $G$ denotes the Catalan's constant and $\beta(4)=\sum_{n=0}^\infty\frac{(-1)^n}{(2n+1)^4}$. Substituting these two results in equation (1) gives: $$\int_0^1 \frac{\log^2(x)\tanh^{-1}(x)}{1+x^2}dx=\beta(4)-\frac{\pi^2}{12}G \tag{4}$$
Proof sketch of integrals (2) and (3) : (Please see the above links for a more detailed answer)
The idea behind evaluating (2) and (3) is breaking them down into Euler Sums. Using the taylor series expansion $\tan^{-1}(x)=\sum_{n=0}^\infty\frac{(-1)^n x^{2n+1}}{2n+1}$ and integrating term-wise, we obtain the following relations:
\begin{align*} \int_0^1\frac{\log(x)\tan^{-1}(x)\tanh^{-1}(x)}{x}dx &= -\log(2)\frac{\pi^3}{32}-\frac{1}{2}\sum_{n=0}^\infty \frac{(-1)^n}{(2n+1)^3}\left( \gamma+\psi_0(n+1)\right) \\ &\;+\frac{1}{4}\sum_{n=0}^\infty \frac{(-1)^n \psi_1(n+1)}{(2n+1)^2} \tag{5}\\ \int_0^1 \frac{\log^2(x)\tan^{-1}(x)}{1-x^2}dx &=-\frac{1}{8}\sum_{n=0}^\infty\frac{(-1)^n\psi_2(n+1)}{2n+1}\tag{6} \end{align*}
These Euler Sums can be evaluated using the techniques shown in the paper "Euler Sums and Contour Integral Representations" by Philippe Flajolet and Bruno Salvy. Here is it's link. \begin{align*} \sum_{n=0}^\infty\frac{(-1)^n\psi_2(n+1)}{2n+1} &= 8\beta(4)+\frac{\pi^2}{3}G-\frac{7\pi}{2}\zeta(3) \\ \sum_{n=0}^\infty\frac{(-1)^n\psi_1(n+1)}{(2n+1)^2} &= 6\beta(4)+\frac{\pi^2}{4}G-\frac{7\pi}{4}\zeta(3) \\ \sum_{n=0}^\infty \frac{(-1)^n\left( \gamma+\psi_0(n+1)\right)}{(2n+1)^3} &= 3\beta(4)-\frac{7\pi}{16}\zeta(3)-\frac{\pi^3}{16}\log(2) \end{align*} Substituting these into equations (5) and (6) gives us the integrals (2) and (3).
A related integral
Using similar techniques, we can show that $$\displaystyle \int_0^1 \frac{\log^2(x)\tan^{-1}(x)}{x\left(1-x^2 \right)}dx=\beta(4)+\frac{7\pi \zeta(3)}{64}+\frac{\pi^3 \log(2)}{32}$$