By transforming to polar coordinates, show that
$$\int_{0}^{1} \int_{0}^{x}\frac{1}{(1+x^2)(1+y^2)} \,dy\,dx$$
Is equal to
$$ \int_{0}^{\pi/4}\frac{\log(\sqrt{2}\cos(\theta))}{\cos(2\theta)} d\theta$$
I have tried the standard $x=r\cos(\theta)$ etc. And can easily see that the limits for theta is $0,\pi/4$ with a sketch. However, I'm struggling to compute the integral that comes out, which ranges $r$ from $r=0$ to $r=\frac{1}{\cos{\theta}}$. Also, I can integrate this explicitly using tan substitution but does that help?
You have correctly converted the region of integration and thus we have:
\begin{align} I &= \int_0^{\frac{\pi}{4}}\int_1^{\sec(\theta)} \frac{1}{\left(1 + r^2\cos^2(\theta)\right)\left(1 + r^2\sin^2(\theta)\right)} \cdot r\:dr\:d\theta \\ \end{align}
Applying a partial fraction decomposition we arrive at: \begin{align} I &= \int_0^{\frac{\pi}{4}}\int_1^{\sec(\theta)} \frac{1}{\left(1 + r^2\cos^2(\theta)\right)\left(1 + r^2\sin^2(\theta)\right)} \cdot r\:dr\:d\theta \\ &= \int_0^{\frac{\pi}{4}}\int_1^{\sec(\theta)} \frac{1}{\cos^2(\theta) - \sin^2(\theta)}\left[\frac{\cos^2(\theta)}{1 + r^2\cos^2(\theta)} - \frac{\sin^2(\theta)}{1 + r^2\sin^2(\theta)} \right] r\:dr\:d\theta \\ &= \int_0^{\frac{\pi}{4}}\frac{1}{\cos^2(\theta) - \sin^2(\theta)}\int_1^{\sec(\theta)}\left[\frac{r\cos^2(\theta)}{1 + r^2\cos^2(\theta)} - \frac{r\sin^2(\theta)}{1 + r^2\sin^2(\theta)} \right]\:dr\:d\theta \\ &=\int_0^{\frac{\pi}{4}}\frac{1}{\cos^2(\theta) - \sin^2(\theta)}\cdot\frac{1}{2}\bigg[\ln\left|1 + r^2\cos^2(\theta) \right| + \ln\left|1 + r^2\sin^2(\theta) \right| \bigg]_1^{\sec(\theta)} \:d\theta \end{align}
You okay from here?
Edit: The rest of the solution:
\begin{align} I &= \int_0^{\frac{\pi}{4}}\frac{1}{\cos^2(\theta) - \sin^2(\theta)}\cdot\frac{1}{2}\bigg[\ln\left|1 + r^2\cos^2(\theta) \right| + \ln\left|1 + r^2\sin^2(\theta) \right| \bigg]_1^{\sec(\theta)} \:d\theta \\ &= \int_0^{\frac{\pi}{4}}\frac{1}{\cos^2(\theta) - \sin^2(\theta)}\cdot\frac{1}{2}\bigg[\ln(2) + \ln\left|\sec^2(\theta) \right| \bigg] \:d\theta \\ &= \int_0^{\frac{\pi}{4}}\frac{1}{\cos(2\theta)}\cdot\frac{1}{2}\ln\left|2\sec^2(\theta)\right|\:d\theta\int_0^{\frac{\pi}{4}}\frac{\ln\left| \sqrt{2}\sec(\theta)\right|}{\cos(2\theta)}\:d\theta \end{align}