The question is to change the cartesian form to the corresponding polar form: $$\int_0^a\int_y^a{\frac{x^2\,dx\,dy}{\sqrt{x^2+y^2}}}$$ The limit when applied in the format $\theta =0$ and $\theta = \frac{\pi}{4}$ and $r=a$ and $r=a\sec(\theta)$ The integral evaluates to $$I=\frac{a^3}{3}\log(1+\sqrt{2})$$
But when the radius limit is changed to $r=a$ and $r=a\sqrt{2}$
the integral evaluates to $$I=\frac{a^3\pi}{24}(2\sqrt{2}-1)$$ is the answer same as above?
A quick check on a calculator, using $a=1$, shows your expressions for $I$ are different. The first gives the value $0.293791$ while the second gives the value $0.239341$. The first one is correct, the second is wrong. The first is also equivalent to $\frac 13\operatorname{sinh}^{-1}(1)$, by the way.
You do not give details on how you found your answers, but I do see that your stated limits on $r$ are wrong. Here is the region of integration:
The limits on $r$ should be $0\le r\le a\sec\theta$, but you stated $a\le r\le a\sec\theta$ the first time and $a\le r\le a\sqrt 2$ the second time. Both statements are wrong. Your limits on theta, $0\le\theta\le\frac{\pi}4$, are correct. As I said, your first value is correct.