The given integral is: $$\int_\frac{-\sqrt3}2^\frac{\sqrt3}2\int_\frac{1}2^\sqrt{1-x^2}\int_\sqrt{x^2+y^2}^{2-x^2-y^2} \frac{x}y dzdydx$$
I converted it as: $$\int_0^\pi \int_0^1\int_r^{2-r} \cot\theta r dzdrd\theta$$
We were only asked to set up the integral and not solve it but I tried to check what the answer to this would be and it will be undefined. What did I do wrong?
You are wrong about the values that $\theta$ can take: it can take any value in $\left[\frac\pi6,\frac{5\pi}6\right]$. For each such $\theta$, the values that $r$ can take go from $\frac{\cos\theta}2$ to $1$. Finally, the values that $z$ can take go from $r$ to $2-r^2$. So, you should have got$$\int_{\pi/6}^{5\pi/6}\int_{\csc(\theta)/2}^1\int_r^{2-r^2}\,\mathrm dz\,\mathrm dr\,\mathrm d\theta$$