Calculate the iterated integral
$$\iint\limits_{D}\sqrt{xy(1-x-y)}\mathrm{d}x\mathrm{d}y,$$ where the domain is $D=\left\{(x,y): x\geq0, y\geq0, x+y\leq1\right\}$
I think the range is $0\leq x\leq1$ and $0\leq y\leq{1-x}$. Is it correct?
I am getting stuck in this problem. Should I integrate by parts, or is there any other way to solve it? If I have to substitute, then what should I substitute? Please anyone help me solving it. Thanks in advance.


Write $x=u^2,\,y=v^2$ so $D=\{(u,\,v)|u,\,v\ge0,\,u^2+v^2\le1\}$ is the first quadrant of the centred-at-$O$ unit circle. Note that $x^{1/2}dx=2u^2du$. Switching to polar coordinates, your integral is$$\iint_D4u^2v^2\sqrt{1-u^2-v^2}dudv=\int_0^{\pi/2}4\cos^2\theta\sin^2\theta d\theta\int_0^1r^5\sqrt{1-r^2}dr.$$I've absorbed the factor of $4$ into the $\theta$ integral so it's$$\int_0^{\pi/2}\sin^22\theta d\theta=\frac{\pi}{4}$$(the integrand famously averages to $\frac12$), while with $t=1-r^2$ the $r$ integral is $$\frac12\int_0^1t^{1/2}(1-t)^2dt=\frac12\int_0^1\left(t^{1/2}-2t^{3/2}+t^{5/2}\right)dt=\frac13-\frac25+\frac17=\frac{8}{105}.$$So our final result is $\frac{2\pi}{105}$.