I have tried to many different ways and I also end up with a tan inside an arctan which I do not know how to simplify. Please suggest how I can solve this. Am I missing something simple or is it quite tricky and need some sort of manipulation that I am missing.
On
Take $ a = b^2$ for convenience and suppose,
$$ \begin{align} y & = \frac{\cos^24x+1}{a\cos^22x+1}\\ &= \frac{(2\cos^2{2x}-1)^2+1}{b^2\cos^22x+1}\\ &= \frac{4\cos^4{2x}-4\cos^2{2x}+2}{b^2\cos^22x+1}\\ &=\frac{4\cos^4{2x}}{b^2\cos^22x+1} -\frac{4\cos^2{2x}}{b^2\cos^22x+1} +\frac{2}{b^2\cos^22x+1}\\ &=A-B+C \end{align} $$
Now, $$ \begin{align} A &= \frac{4\cos^4{2x}}{b^2\cos^22x+1}\\ &=\frac{4}{b^4} \left( \frac{b^4\cos^4{2x}}{b^2\cos^22x+1} \right)\\ &=\frac{4}{b^4} \left( \frac{b^4\cos^4{2x}-1+1}{b^2\cos^22x+1} \right)\\ &=\frac{4}{b^4} \left( \frac{((b\cos2x)^2+1)((b\cos2x)^2-1)}{b^2\cos^22x+1} +\frac{1}{b^2\cos^22x+1} \right)\\ &=\frac{4}{b^4} \left( (b\cos{2x})^2-1 +\frac{1}{b^2\cos^22x+1} \right)\\ &=\frac{4}{b^2}\cos^2{2x}+\frac{4}{b^4} \left( \frac{1}{b^2\cos^22x+1} \right)-\frac{4}{b^4}\\ \text{or} \ A&=\frac{4}{a}\cos^2{2x}+\frac{4}{a^2} \left( \frac{1}{a\cos^22x+1} \right)-\frac{4}{a^2} \qquad(1) \end{align} $$
Simplifying B, $$ \begin{align} B &= \frac{4\cos^2{2x}}{b^2\cos^22x+1}\\ &= \frac{4}{b^2} \left( \frac{(b\cos{2x})^2}{b^2\cos^22x+1} \right)\\ &= \frac{4}{b^2} \left( \frac{(b\cos{2x})^2+1-1}{(b\cos{2x})^2+1} \right)\\ &= \frac{4}{b^2} \left(1 - \frac{1}{(b\cos{2x})^2+1} \right)\\ &= \frac{4}{b^2} -\frac{4}{b^2}\left(\frac{1}{b^2\cos^2{2x}+1} \right)\\ \text{or} \ B &= \frac{4}{a} -\frac{4}{a}\left(\frac{1}{a\cos^2{2x}+1} \right) \qquad(2)\\ \end{align} $$
Now,
$$ \begin{align} A-B+C &= \frac{4}{a}\cos^2{2x}+\frac{4}{a^2} \left( \frac{1}{a\cos^22x+1} \right)-\frac{4}{a^2} -\frac{4}{a} +\frac{4}{a}\left(\frac{1}{a\cos^2{2x}+1} \right) +\frac{2}{b^2\cos^22x+1}\\ \text{or}\ y&= \frac{4}{a}\cos^2{2x} + \left( \frac{4}{a^2} + \frac{4}{a} +2 \right)\frac{1}{a\cos^2{2x}+1} -\left( \frac{4}{a^2} + \frac{4}{a} \right) \qquad(3)\\ \end{align} $$ Now, $$ \int^\pi_{-\pi}\cos^2{2x}\,dx = {\pi} \qquad(4) $$ and, $$\begin{align} \int^{\pi}_{-\pi}\frac{1}{a\cos^2{2x}+1}\,dx &= 8\int^{\pi/4}_{0}\frac{1}{a\cos^2{2x}+1}\,dx \\ &= 8\int^{\pi/4}_{0}\frac{\sec^2{2x}}{a+\sec^2{2x}}\,dx\\ &= 8\int^{\pi/4}_{0}\frac{\sec^2{2x}}{a+1+\tan^2{2x}}\,dx\\ &= 4\frac{1}{\sqrt{a+1}} {\left[\tan^-1{\left(\frac{\tan{2x}}{\sqrt{a+1}} \right)}\right]}_{0}^{\pi/4}\\ &= \frac{2\pi}{\sqrt{a+1}} \qquad(5) \end{align} $$
Now, $$ \begin{align} \frac{1}{\pi}\int^{\pi}_{-\pi}\frac{\cos^2(4x) +1}{a\cos^2(2x)+1}\,dx &= \frac{1}{\pi}\int^{\pi}_{-\pi}y\,dx\\ &= \frac{1}{\pi} \left({\frac{4}{a}{\pi} + \left(\frac{4}{a^2} +\frac{4}{a}+2 \right)\frac{2\pi}{\sqrt{a+1}} -2{\pi}\left( \frac{4}{a^2} + \frac{4}{a} \right)} \right)\\ \end{align} $$ Therefore, as per the problem statement, $$ {\frac{4}{a} + \left(\frac{4}{a^2} +\frac{4}{a}+2 \right)\frac{2}{\sqrt{a+1}} -2\left( \frac{4}{a^2} + \frac{4}{a} \right)} =a $$ A multiplication of ${a^2}{\sqrt{a+1}} $ on both sides and then squaring both side will yield $$ \begin{align} & a^7 +a^6+8a^5+8a^4-32a^3-48a^2=0\\ & \Rightarrow a^5+a^4+8a^3+8a^2-32a =48\\ & \Rightarrow \sqrt{\sqrt{\frac{1}{3}(a^5+a^4+8a^3+8a^2-32a)}} \Rightarrow \sqrt{\sqrt{\frac{48}{3}}} = 2. \end{align} $$
By symmetry we can reduce the interval to $[0,\frac{\pi}4]$, the coefficient before the integral sign becomes $8$.
$$I=\frac 8{\pi}\int_0^{\frac{\pi}4}\dfrac{\cos(4x)^2+1}{a\cos(2x)^2+1}dx$$
Then by substitution $u=2x$:
$$I=\frac 4{\pi}\int_0^{\frac{\pi}2}\dfrac{\cos(2u)^2+1}{a\cos(u)^2+1}du$$
The reason behind reducing the interval is to be able to have a bijective change $t=\tan(u)$:
We expand $\cos(2u)$ and replace $\cos(u)^2$ by $\frac 1{1+t^2}$:
$$I=\frac 8{\pi}\int_0^{\infty}\dfrac{t^4+1}{(t^2+a+1)(t^2+1)^2}du$$
The next part is partial fraction decomposition and integration in arctan, it is not difficult but tedious, so I skip it and jump to the result:
$$\dfrac {\tfrac{a^2+2a+2}{a^2}}{t^2+a+1}-\dfrac {\frac{2a+2}{a^2}}{t^2+1}+\dfrac {\frac 2a}{(t^2+1)^2}$$
$$I=4\times\frac{(a^2+2a+2)-\sqrt{1+a}(a+2)}{a^2\sqrt{1+a}}$$
We now have to solve $I=a$ :
$4(a^2+2a+2)-4\sqrt{1+a}(a+2)=a^3\sqrt{1+a}\iff 4(a^2+2a+2)=\sqrt{1+a}(a^3+4a+8)$
We square both sides and simplify to:
$$a^7+a^6+8a^5+8a^4-32a^3-48a^2=0$$
And since $a\neq 0$ we are glad to arrive to the desired expression: