Using spherical coordinates, express the triple integral of $f$ over the region $$x^2+y^2+z^2\le 2ay\le 2a^2$$ for $a>0$.
I tried $$\int_{0}^{\pi}d{\theta}\int_{0}^{\pi}\sin \phi d{\phi} \int_{0}^{2a \sin \theta \sin \phi} f \rho^2d{\rho}$$ but this gives the full sphere instead of the region described in the problem.
The inequality $x^2+y^2+z^2\le 2ay$ can be written as $$x^2+(y-a)^2+z^2\le a^2$$ which is a full sphere of radius $a$ centered at $(0,a,0)$. The second inequality $2ay\le 2a^2$ is equivalent to $y\leq a$. So the sphere is cut in two full hemispheres by the plane $y=a$ (which is through the center) and the domain is the hemisphere which contains the origin.
Hence, I suggest using these modified spherical coordinates: $$x=\rho\sin(\phi)\cos(\theta), y=a+\rho\cos(\phi), z=\rho\sin(\phi)\sin(\theta)$$ with $\rho\in [0,a]$, $\theta\in [0,2\pi]$ and $\phi\in [\pi/2,\pi]$.
Ordinary spherical coordinates give you: $$x=\rho\sin(\phi)\cos(\theta), y=\rho\sin(\phi)\sin(\theta), z=\rho\cos(\phi),$$ with $\theta\in [0,\pi]$ and $\phi\in [0,\pi]$ and $\rho\in [0,a\min\left(\frac{1}{\sin(\phi)\sin(\theta)},2\sin(\phi)\sin(\theta)\right)]$.