Evaluate $\int\limits_0^1x(\tan^{-1}x)^2~\textrm{d}x$
My Attempt
Let, $\tan^{-1}x=y\implies x=\tan y\implies dx=\sec^2y.dy=(1+\tan^2y)dy$ $$ \begin{align} &\int\limits_0^1x(\tan^{-1}x)^2dx=\int\limits_0^{\pi/4}\tan y.y^2.(1+\tan^2y)dy\\ &=\int\limits_0^{\pi/4}\tan y.y^2dy+\int\limits_0^{\pi/4}\tan^3y.y^2dy\\ &=\bigg[y^2.\log|\sec y|-\int2y.\log|\sec y|dy\bigg]+\int_0^{\pi/4}\tan^3y.y^2dy\\ &=\bigg[y^2.\log|\sec y|-y^2.\log|\sec y|+\int\frac{\tan y\sec y}{\sec y}y^2dy\bigg]+\int_0^{\pi/4}\tan^3y.y^2dy\\ \end{align} $$ How do I proceed further and solve the integration?
The OP's initial change of variables will lead to an answer with a couple of integrations by parts and an intervening trig identity:
$$\begin{align} \int_0^1x(\arctan x)^2dx &=\int_0^{\pi/4}y^2\tan y\sec^2y\,dy\\ &={1\over2}y^2\tan^2y\,\Big|_0^{\pi/4}-\int_0^{\pi/4}y\tan^2y\,dy\quad(u=y^2,\,\, dv=\tan y\sec^2y\,dy)\\ &={1\over2}\left(\pi\over4\right)^2-\int_0^{\pi/4}y(\sec^2y-1)\,dy\quad\text{(trig identity)}\\ &={\pi^2\over32}+\int_0^{\pi/4}y\,dy-\int_0^{\pi/4}y\sec^2y\,dy\\ &={\pi^2\over32}+{\pi^2\over32}-\left(y\tan y\,\Big|_0^{\pi/4}-\int_0^{\pi/4}\tan y\,dy \right)\quad(u=y,\,\, dv=\sec^2y\,dy)\\ &={\pi^2\over16}-{\pi\over4}-\left(\ln\cos y\,\Big|_0^{\pi/4} \right)\\ &={\pi^2\over16}-{\pi\over4}-\ln(1/\sqrt2)+\ln(1)\\ &={\pi^2\over16}-{\pi\over4}+{1\over2}\ln2 \end{align}$$
Remarks: Once the change of variable left you integrating $y^2\times$trig($y$), you pretty much have to hammer away at the $y^2$ with integration by parts, first with $u=y^2$, then with $u=y$. The OP did a seemingly sensible thing, breaking $\int y^2(\tan y+\tan^3y)dy$ into separate pieces, $\int y^2\tan y\,dy$ and $\int y^2\tan^3y\,dy$. The problem is, neither of these pieces, by itself, has a nice antiderivative. Integration by parts on $\int y^2\tan y\,dy$, with $u=y^2$ and $dv=\tan y\,dy$, leaves $\int y\ln\sec y\,dy$, as the OP found, but at that point you're stuck. What the OP did was, in effect, to undo the first integration by parts, by letting $u=\ln\sec y$ and $dv=y\,dy$. The take-home lesson here is that trig identities and integration by parts are wonderful tools, but sometimes they'll lead you to dead ends, so you have to be ready, sometimes, to give up and start anew. Practice won't make perfect, but it will help you develop a nose for which pathways are likely to pay off.