$S$ is the region that is enclosed by the unit sphere $x^2+y^2+z^2=1$ and the cone $z=(x^2+y^2)^{1/2}$.
So I decided to use cylindrical coordinates first:
Define $G(r,\theta,z)=(rcos\theta,rsin\theta, z)$ and $\det DG=r$
This means our first condition becomes $r^2+z^2=1$ and second one becomes $z=r$.
$Volume(S)=\int\int\int_SrdV$. I am not sure how to find the limits of integration.
I know $\theta\in[0,2\pi]$. And we know $z=r \implies 2r^2=1\implies r= \pm\frac{1}{\sqrt2} \implies -\frac{1}{\sqrt{2}}\leq r \leq\frac{1}{\sqrt{2}}...?$ And $z=\pm\sqrt{1-r^2}?$ I'm not sure if it's supposed to be the other way around for $r$ and $z$ and how are you supposed to know?
The mass is defined to be $\int\int\int_S \rho(x)dV$ but I'm not sure what's $\rho(x)$... But I know how to calculate the centroid.
The limits for $\theta$ are correct. For $r$, as it has to be positive, are $0\leq r\leq\dfrac{1}{\sqrt{2}}$. Now, the region of integration is an inverted cone with its vertex at the origin (the equation considers only the positive square root) and the sphere as its "ceiling", so, we have different formulas for the lower limit and for the upper one: $r\leq z\leq \sqrt{1-r^2}$
$$V(S)=\int_0^{2\pi}\int_0^{1/\sqrt{2}}\int_r^{\sqrt{1-r^2}}r\mathbb dz\,\mathbb dr\,\mathbb d\theta$$
For the centroid we have to evaluate three integrals, one for each coordinate $C=(C_r,C_{\theta},C_z)$ but we can take advantage of the region's symmetry of rotation around the $z$ axis. The centroid is in some point of this axis $C=(0,0,C_z)$, with,
$$C_z=\int_0^{2\pi}\int_0^{1/\sqrt{2}}\int_r^{\sqrt{1-r^2}}zr\mathbb dz\,\mathbb dr\,\mathbb d\theta$$