$\displaystyle \int_0^1 \tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2} = \frac{\pi^2}{8}$
I had been told that $x =\displaystyle \frac{1-y}{1+y}$ solves it, but it takes me back to where I started.
$\displaystyle \int_0^1 \tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2} = \frac{\pi^2}{8}$
I had been told that $x =\displaystyle \frac{1-y}{1+y}$ solves it, but it takes me back to where I started.
On
$$\int_0^1 \tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2} $$ $$=\int_0^1 \tan^{-1}\left(\frac{3(1+x)}{2-1-2x-x^2}\right) \frac{dx}{1+x^2} $$ $$=\int_0^1 \tan^{-1}\left(\frac{3(1+x)}{2-(1+x)^2}\right) \frac{dx}{1+x^2} $$ $$=\int_0^1 \tan^{-1}\left(\frac{\frac{3}{2}(1+x)}{1-\frac{1}{2}(1+x)^2}\right) \frac{dx}{1+x^2} $$ $$=\int_0^1 \tan^{-1}\left(\frac{(1+x)+\frac{1}{2}(1+x)}{1-(1+x)\cdot \frac{1}{2}(1+x)}\right) \frac{dx}{1+x^2} $$ $$=\int_0^1 \left[\tan^{-1}(1+x)+\tan^{-1}\{\frac{1}{2}(1+x)\}\right] \frac{dx}{1+x^2} $$ $$=\int_0^1 \left[\tan^{-1}(1+x)+\tan^{-1}\{\frac{1}{2}(1+x)\}\right] d(\tan^{-1}x) $$
Hope this helps.
On
In fact, the correct answer is 0. As @JimmyK4542 computed, we know from the substitution $x \mapsto (1-x)/(1+x)$ that
$$ I = \int_{0}^{1} \arctan\left( \frac{3(1+x)}{1-2x-x^2} \right) \, \frac{dx}{1+x^2} = \int_{0}^{1} \arctan\left( - \frac{3(1+x)}{1-2x-x^2} \right) \, \frac{dx}{1+x^2}. $$
But the identity $\arctan(-x) = -\arctan x$ is true for any $x \in \Bbb{R}$. Consequently
$$ I = -I $$
and hence $I = 0$. So why we have vanishing quantity here? This is because the function satisfies
$$ x \mapsto \arctan\left(\frac{3(1+x)}{1-2x-x^2}\right) \quad \text{is} \quad \begin{cases} \text{positive when } 0 < x < \sqrt{2}-1 \\ \text{negative when } \sqrt{2}-1 < x < 1. \end{cases} $$
So the graph of this function looks like this:
Somehow magically, the contribution from the positive part and that from the negative part cancel out each other. This observation is also supported by numerical calculation:
Using the suggested substitution $x = \dfrac{1-y}{1+y}$, we get
$\dfrac{3(1+x)}{1-2x-x^2} = -\dfrac{3(1+y)}{1-2y-y^2} \ \ \ \ \ \ \ \ \dfrac{1}{1+x^2} = \dfrac{(1+y)^2}{2(1+y^2)} \ \ \ \ \ \ \ \ dx = \dfrac{-2}{(1+y)^2}\,dy$.
and thus,
$I = \displaystyle \int_0^1 \tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2}$
$= \displaystyle \int_1^0 \tan^{-1}\left(-\frac{3(1+y)}{1-2y-y^2}\right)\cdot \dfrac{(1+y)^2}{2(1+y^2)} \cdot \dfrac{-2}{(1+y)^2}\,dy$
$= \displaystyle \int_0^1 \tan^{-1}\left(-\frac{3(1+y)}{1-2y-y^2}\right) \frac{dy}{1+y^2}$
$= \displaystyle \int_0^1 \tan^{-1}\left(-\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2}$.
Now, assuming we use a definition of $\arctan$ whose range is $[0,\tfrac{\pi}{2}) \cup (\tfrac{\pi}{2},\pi)$ instead of the usual $(-\tfrac{\pi}{2},\tfrac{\pi}{2})$ then we have $\arctan(A) + \arctan(-A) = \pi$. Hence,
$2I = I + I = $ $\displaystyle \int_0^1 \tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2} + \int_0^1 \tan^{-1}\left(-\frac{3(1+x)}{1-2x-x^2}\right) \frac{dx}{1+x^2}$
$= \displaystyle \int_0^1 \left[\tan^{-1}\left(\frac{3(1+x)}{1-2x-x^2}\right)+\tan^{-1}\left(-\frac{3(1+x)}{1-2x-x^2}\right)\right]\frac{dx}{1+x^2}$
$= \displaystyle\int_0^1 \pi \dfrac{dx}{1+x^2}$
$= \pi \cdot \dfrac{\pi}{4} = \dfrac{\pi^2}{4}$.
Since $2I = \dfrac{\pi^2}{4}$, we have $I = \dfrac{\pi^2}{8}$.
Remark: If we use the usual definition of $\arctan$ whose range is $(-\tfrac{\pi}{2},\tfrac{\pi}{2})$, then we get $I = 0$, which is what Wolfram Alpha suggests as the answer.