Rewrite $$\int_0^1 \int_{-\sqrt{y-y^2}}^{0}(x^2+y^2)\,\text{d}x \, \text{d}y$$ in polar.
We see that $-\sqrt{y-y^2}\leq x\leq 0$.
If $x=-\sqrt{y-y^2}$, then $x^2=y-y^2\to x^2+y^2-y=0\to \color{red}{x^2+(y-\dfrac{1}{2})^2=\dfrac{1}{4}}$

Clearly $0\leq\theta\leq\pi$. Also, $x^2+y^2=y\to r^2\sin^2(\theta)+ r^2\cos^2(\theta)=r\sin(\theta)\to r^2=r\sin(\theta)\to\color{red}{r=\sin(\theta)}$
Therefore, $0\leq r \leq \sin(\theta)$
Thus the new integral is
$$\int^\pi_0 \int^{\sin(\theta)}_0 r^2 r \, \mathrm{d}r \, \mathrm{d}\theta=\frac{3\pi}{32}$$
But wolfram alpha says the original integral in cartesian coordinates evaluates to $\dfrac{3\pi}{64}$?

$x$ takes only non-positive value. $\theta$ doesn't go from $0$ to $\pi$.
$\theta$ should go from $\frac{\pi}{2}$ to $\pi$.