Here is the problem:
Use divergence theorem to compute 
$S$ is boundary of solid bounded above cone $z = \sqrt{x^2+y^2}$ below sphere $x^2+y^2+z^2 = 1$.
Here is my solving process:
When setting up the integral, the bounds I chose are
- $ 0 <r < \frac{\sqrt{2}}{2}$ , upper bound from 2 solid's intersection
- $ 0< \theta <2\pi$
- $ 0 < z <\sqrt{1-r^2}$, upper bound comes from equation of sphere
Then the $\nabla \cdot F = x^2+y^2+z^2$, and $dV = rdzdrd\theta$.
Here is my confusion:
The result I've got is incorrect, and the only problem I can think of is the lower bound of $z$. It might be $r$ (coming from the equation of cone) instead of $0$ but why? The bounded solid E starts from origin, isn't it? Hope someone will answer my doubt.
Plus, are there any other problems in my solving process? Thanks!
Your divergence is correct. The problem as you mentioned is with the lower bound of $z$. In cylindrical coordinates, the integral should be set up as -
$\displaystyle \int_0^{2\pi} \int_0^{1/ \sqrt 2}\int_r^{\sqrt{1-r^2}} r(r^2+z^2) \, dz \, dr \, d\theta$
On your question as to why the lower bound should be $r$ and not $0$, first deal with the cone with base as $z = \frac{1}{\sqrt2} \,$ and vertex at the origin, without the spherical cap on top. For a given $r$, can $z$ freely vary between $0$ and $1 / \sqrt2$? That will give you the volume of the cylinder instead of the cone. In a cone, $z$ can vary from the base to the points on its rays, ($r \leq z \leq \frac{1}{\sqrt2}$). Now the upper bound of $z$ changes because we are just adding the spherical cap on top.
In spherical coordinates which is easier in this case,
$\displaystyle \int_0^{2\pi} \int_0^{\pi/4}\int_0^1 r^4 \sin \phi \, dr \, d\phi \, d\theta$