I have a hard time wrapping my mind around what boundaries that I should use to evaluate the following integral: $$\iint_M xy\,dx\,dy$$ in the region $M: \{(x,y) \in \mathbb{R}: |x|\le 1 , |y|\le 1, x^2+y^2 \ge 1\}$
From what I understand, I am supposed to find the volume inside the circle up to the $f(x,y)=xy.$ First, I think that the equation for the circle should be less or equal to $1.$ One of the boundaries should be $-1$ to $1$ and the other one I can't seem to understand. I've tried going from $-1$ to $y\le\sqrt{1-x^2}$ . Evaluating this way I got $0.$ Is the info about the area correctly given or is my assumption right? And what boundaries should I use?

Here's the way I would go about solving this problem. I would note that it's easier to integrate on the circle $x^2+y^2\le 1$ than outside it, and it's also easier to integrate in the square $|x|\le 1, |y|\le 1$ than over what's inside the square but outside the circle. Therefore, I would write $C=\{(x,y):x^2+y^2\le 1\}$ and $B=\{(x,y):|x|\le 1, |y|\le 1\}.$ Then we can write $$\iint_M xy\,dx\,dy=\iint_{B}xy\,dx\,dy-\iint_Cxy\,dA. $$ So for the first integral, we just have \begin{align*} \int_{-1}^1\int_{-1}^1xy\,dx\,dy&=\int_{-1}^1x\,dx\cdot\int_{-1}^1y\,dy\\ &=\left(\frac{x^2}{2}\Bigg|_{-1}^1\right)^{\!2}\\ &=0. \end{align*} For the second integral, we switch to polar thus: \begin{align*} \iint_Cxy\,dA&=\int_{0}^{2\pi}\int_0^1(\underbrace{r\cos(\theta)}_{x})(\underbrace{r\sin(\theta)}_{y})\,\underbrace{r\,dr\,d\theta}_{dA}\\ &=\int_0^{2\pi}\cos(\theta)\sin(\theta)\,d\theta\cdot\int_0^1r^3\,dr\\ &=\left(-\frac{\cos^2(\theta)}{2}\right)\Bigg|_0^{2\pi}\cdot\left(\frac{r^4}{4}\right)\Bigg|_0^1\\ &=0\cdot \frac14\\ &=0. \end{align*} Therefore, the original integral is zero.
We can check this result by dividing up $M$ according to quadrants. Note that the integrand $f(x,y)=xy$ is positive in the first and third quadrants, and negative in the second and fourth quadrants. The hunch, therefore, is that the result in the first and second quadrants will be zero, and the result in the third and fourth quadrants will also be zero. Let $M^+=\{x\in M:y\ge 0\}.$ So this covers the first and second quadrants. We have \begin{align*} \iint_{M^+}xy\,dx\,dy&=\iint_{M^+}xy\,dy\,dx\\ &=\int_{-1}^1\int_{\sqrt{1-x^2}}^{1} xy\,dy\,dx\\ &=\int_{-1}^1 x \left(\frac{y^2}{2}\right)\Bigg|_{\sqrt{1-x^2}}^{1}dx\\ &=\int_{-1}^1 x \left(\frac12-\frac{1-x^2}{2}\right)dx\\ &=\frac12\int_{-1}^1 x^3\,dx\\ &=0, \end{align*} as expected. You'll find the same occurs for the third and fourth quadrants.