Compute the gravitational attraction on a unit mass at the origin due to the mass (of constant density) occupying the volume inside the sphere $r = 2a$ and above the plane $z=a$. Use spherical coordinates.
So I know the function should be $$(G/r^2) dM$$ What are the limits of integration? What should the integral look like?
While Rafa gave a good answer, I just wanted to expand a bit on the derivation here. That way you're less likely to get lost. Let me know if I need to explicate further on some part.
A couple of physics formulas necessary for this:
Given that this is a spherical cap
where the radius is $2a$ and the height of cap is $a$, we see that the polar angle $\theta$ varies over $0$ to whatever $\theta$ is at the bottom of the cap. To figure that out, notice that a triangle is made (look at the shape in the picture above, but ignore the letters -- those don't correspond to this exercise) where the hypotenuse is $2a$ and the adjacent side is has length $a$. Hence $\cos(\theta) = \dfrac{a}{2a} = \dfrac 12 \implies \theta = \dfrac{\pi}{3}$.
The azimuthal angle $\varphi$ goes all the way around the circle -- i.e. it varies over $0$ to $2\pi$. And $r$ is going to be a bit trickier -- it'll be a function of $\theta$. When $\theta = 0$, $r$ can vary over $a$ to $2a$, but when $\theta=\frac{\pi}{4}$, $r$ can only be $2a$. In general, for a given $\theta$, $r$ can go no lower than the plane $z=a$ -- which in spherical coordinates is $\rho\cos(\theta) =a$. I.e. $r$ varies over $a\sec(\theta)$ to $2a$.
Now that we have our bounds, we can set up our integral:
$$\begin{align}\mathbf F &= \iiint_\text{cap} d\mathbf F \\ &= \iiint_\text{mass of cap} \mathbf {\hat r}\frac{G(1)dM}{r^2} \\ &= G\rho \int_0^{2\pi}\int_0^{\pi/3}\int_{a\sec(\theta)}^{2a} \mathbf {\hat r}\frac{r^2\sin(\theta)\ drd\theta d\varphi}{r^2} \\ &= G\rho \int_0^{2\pi}\int_0^{\pi/3}\int_{a\sec(\theta)}^{2a} \left(\cos(\varphi)\sin(\theta)\mathbf {\hat x} + \sin(\varphi)\sin(\theta)\mathbf {\hat y} + \cos(\theta)\mathbf {\hat z}\right)\sin(\theta)\ drd\theta d\varphi \\ &= G\rho\mathbf {\hat x}\int_0^{2\pi}\int_0^{\pi/3}\big(2a-a\sec(\theta)\big) \cos(\varphi)\sin(\theta)^2\ d\theta d\varphi \\ &\ \ \ \ + G\rho\mathbf {\hat y}\int_0^{2\pi}\int_0^{\pi/4}\big(2a-a\sec(\theta)\big)\sin(\varphi)\sin(\theta)^2\ d\theta d\varphi \\ &\ \ \ \ + G\rho\mathbf {\hat z}\int_0^{2\pi}\int_0^{\pi/4}\big(2a-a\sec(\theta)\big)\cos(\theta)\sin(\theta)\ d\theta d\varphi \\ &= 0\mathbf {\hat x} + 0\mathbf{\hat y} + G\rho\mathbf {\hat z}\left(a\frac {\pi}2\right) \\ &= \frac{\pi G\rho a}{2}\mathbf{\hat z} \end{align}$$
But it might be useful to go a step further. The exercise doesn't give us a value for the density or mass of the spherical cap. However, it may be more convenient to express it in terms of mass than density (feel free to skip this part and just use the above if you don't think your professor wants you to do this). Hence we use the definition $$\rho = \frac{M}{V}$$ to plug in for the $\rho$. That will require us to find the volume of a spherical cap. You can do so with an integral very similar to the one we calculated above ... or you can use the fact that it's been calculated before and just look up the formula. In this case we have $$V = \frac {\pi(a)}{6}\left(3((2a)^2-a^2)+a^2\right) = \frac{5}{3}\pi a^3$$
Thus we have $\rho = \dfrac{m}{\frac{5}{3}\pi a^3} = \dfrac{3m}{5\pi a^3}$ and hence the force on our mass is
$$\mathbf F = \frac{3G m}{10a^2}\mathbf{\hat z}$$