I want to establish the following identity
$$ \int_0^{\pi/3} \arctan (\sqrt{5} \tan \theta) \, d\theta - 2 \int_0^{\pi/6} \arctan (\sqrt{5} \tan \theta) \, d\theta = \frac{\pi^{2}}{30}. \tag{1} $$
by a direct calculation, that is, not relying on some geometry.
Background. This integral is a by-product of the Schläfli's formula for some orthoscheme (special kind of tetrahedron) in the 3-sphere, according to the problem posed by Prof. H. S. M. Coxeter in AMM 95 (1988). He originally asked to find the value of
$$ \int_{1}^{6} \frac{\operatorname{arcsec} t}{(t+2)\sqrt{t+1}} \left[ \frac{1}{\sqrt{t+3}} + 2 \right] \, dt = \frac{2\pi^{2}}{15} $$
without appealing to geometry or the computer. Then another literature shows that it is also written as
$$ \int_{0}^{\frac{\pi}{3}} \arccos\left( \frac{\cos\theta}{1 + 2\cos\theta} \right) \, d\theta = \frac{2\pi^{2}}{15}. $$
(I guess they are related by the volume of the same geometric object, but I'm not sure since the literature is written in French which I can hardly read.) Applying some manipulations to the last integral, we obtain the four times of the left-hand side of $\text{(1)}$.
My trial. Numerical tests show that the constant $\sqrt{5}$ in $\text{(1)}$ is very special; it seems that no other square root of integer (other than 1) gives a rational multiple of $\pi^{2}$. I tried some basic techniques, all of which turned out to be futile.
So I want to post this problem so that everyone can share it.
This doesn't answer the question. But I believe someone can hopefully take it from here and proceed.
Let $$I(a) = \int_{\pi/6}^{\pi/3} \arctan(a\tan(t))dt - \int_0^{\pi/6} \arctan(a\tan(t))dt$$ Note that $$I(1) = \int_{\pi/6}^{\pi/3} tdt - \int_0^{\pi/6} tdt = \left.\dfrac{t^2}2 \right \vert_{\pi/6}^{\pi/3} - \left.\dfrac{t^2}2 \right \vert_{0}^{\pi/6} = \dfrac{\pi^2/9}2 - \dfrac{\pi^2}{36} = \dfrac{\pi^2}{36}$$ We then have $$I'(a) = \int_{\pi/6}^{\pi/3} \dfrac{\tan(t)}{a^2 \tan^2(t)+1}dt - \int_0^{\pi/6} \dfrac{\tan(t)}{a^2 \tan^2(t)+1}dt$$ $$2I'(a) = \int_{\pi/6}^{\pi/3} \dfrac{\sin(2t)}{a^2 \sin^2(t)+\cos^2(t)}dt - \int_{0}^{\pi/6} \dfrac{\sin(2t)}{a^2 \sin^2(t)+\cos^2(t)}dt$$ Setting $\sin^2(t) = x$, we get $\sin(2t)dt = dx$. Hence, $$2I'(a) = \int_{1/4}^{3/4} \dfrac{dx}{(a^2-1)x+1} - \int_{0}^{1/4} \dfrac{dx}{(a^2-1)x+1}$$ Hence, $$2I'(a) = \dfrac{\log(3a^2+1)}{a^2-1} - 2\dfrac{\log(a^2+3)}{a^2-1} + \dfrac{\log(4)}{a^2-1}$$ Integrating the right hand side gives the expressions in terms of $\log$ and polylogarithm function $\text{Li}_2$. Hopefully, this can be simplified further to get to the answer.