$$I=\iint_{D}x^2\mathrm{dx}\mathrm{dy}~~,~~D~:~0\leq x,~0\leq y~~,~~\sqrt{x}+\sqrt{y}\leq1\tag{1}$$
A change of variables is applied here to solve this problem.
$$x=r^4\cos\left(\theta_{}\right)^4\tag{2}$$
$$y=r^4\sin\left(\theta_{}\right)^4\tag{3}$$
$$D~\rightarrow~D'~~:~~0\leq r\leq1~~,~~0\leq\theta_{}\leq\frac{\pi}{2}~~\text{with}~~\left(x,y\right)~\rightarrow~\left(r,\theta_{}\right)\tag{4}$$
What I can't get currently is the new domain of integration D'.
About third inequality of$~D~$, I've done the following operations.
$$\sqrt{x}+\sqrt{y}\leq1\tag{5}$$
$$\sqrt{r^4\cos\left(\theta_{}\right)^4}+\sqrt{r^4\sin\left(\theta_{}\right)^4}\leq1\tag{6}$$
$$\sqrt{\left(r^2\cos\left(\theta_{}\right)^2\right)^2}+\sqrt{\left(r^2\sin\left(\theta_{}\right)^2\right)^2}\leq1\tag{7}$$
$$\left|r^2\cos\left(\theta_{}\right)^2\right|+\left|r^2\sin\left(\theta_{}\right)^2\right|\leq 1\tag{8}$$
$$r^2\cos\left(\theta_{}\right)^2+r^2\sin\left(\theta_{}\right)^2\leq 1\tag{9}$$
$$2r^2\leq 1~~\iff~~r^2\leq\frac{1}{2}~~\impliedby~~0\leq r\leq\frac{1}{\sqrt{2}}\tag{10}$$
My brain has been confused from eqn10.
So at least with$~\sqrt{x}+\sqrt{y}\leq1~$,
as we fix the value of$~y~$to$~0~$,
$$\operatorname{max}\left(x\right)=1\tag{11}$$
is held hence$~x=r^4\cos\left(\theta_{}\right)^4~$with$~0\leq r\leq1~~,~~0\leq\theta_{}\leq\frac{\pi}{2}~$is quite clear for me.
Vice versa with$~y~$.
The confusing part for me is$~0\leq r\leq\frac{1}{\sqrt{2}}~$
I can't get why this$~\frac{1}{\sqrt{2}}~$can be discarded.
In the first place, I can't get why the change of variables of$~x=r^4\cos\left(\theta_{}\right)^4~~,~~y=r^4\sin\left(\theta_{}\right)^4~$is adequte with$~\sqrt{x}+\sqrt{y}\leq1~$
Can anyone explain me about it?
As called out in the other answer, once you fix your mistake you do get $0 \leq r \leq 1$.
While substitution $x = r^4 \cos^4\theta, y = r^4 \sin^4\theta$ works, it is easier to use substitution $u = \sqrt x, v = \sqrt y$
$|J| = 4 uv~$ and $~\sqrt x + \sqrt y \leq 1$ transforms to $u + v \leq 1$. As $x, y \geq 0$, we also have $u, v \geq 0$
The integral simplifies to,
$ \displaystyle 4 \int_0^1 u^5 \left[\int_0^{1-u} v~dv \right] du = \frac{1}{84}$