I have solved this problem. I wish to find out if my solution is correct. I am a bit confuse with the second part of question (b).
Problem: Let S be the surface of a solid $R$ , which lies inside the cylinder: $$x^2+y^2=16$$ and between the plane
where $x=0$ and $z=5$
There is also defined a vector field F by: $$\begin{align}F(x,y)=(-x^3i-y^3j+3z^2k)\end{align}$$
(a) Calculate : $$\iint_{T} F.\hat n\mathrm dS$$
with T = {(x,y,5)$\in$ $\mathbb{R^3}$|$x^2+y^2\le16$}
(b) Calculate DivF and $$\iint_{S} F.\hat n\mathrm dS$$
with n the outward pointing unit normal.
(c) Calculate: $$\iint_{V} F.\hat n\mathrm dS$$
with V = {(x,y,z)$\in$ $\mathbb{R^3}$|$x^2+y^2=16$ and $0\le$z$\le$5} and the unit normal $\hat n$ points out of the solid $R$
Solution: (a) On the top surface of the Cylinder z = 5, $\hat n$ = $\hat k$ $$F.\hat n = [-x^3i +-y^3j +3z^2k].[k]=3z^2 $$ $$\iint_{S} F.\hat n \mathrm dS = \iint_{S} 3z^2\mathrm dS$$ $$\iint_{S} F.\hat n \mathrm dS = \iint_{S} 3(5^2)\mathrm dS$$ $$\iint_{S} F.\hat n \mathrm dS = 75\iint_{S}\mathrm dS$$ The area enclosed by the circle is $\pi$$r^2$ = 16$\pi$ since the radius of the circle is 4. Therefore $$\iint_{S} F.\hat n \mathrm dS = 75(16\pi) = 1200\pi $$
(b) $$DivF = \nabla.F = [i\frac{\partial }{\partial x}+ j\frac{\partial }{\partial y} +k\frac{\partial }{\partial z}].[-x^3i +-y^3j +3z^2k]= -3x^2-3y^2 +6z$$ $$DivF = \nabla.F = -3(x^2+y^2-2z)$$ $$\iint_{S} F.\hat n \mathrm dS =\iiint_{V} \operatorname{div} F dV -\iint_{S_1} F.\hat n \mathrm dS$$ $$ \iint_{S_1} F.\hat n \mathrm dS = 0 $$ since z=0, then $$ \iint_{S} F.\hat n \mathrm dS = \iiint_{V} \operatorname{div} F dV $$
How do I get $\hat n$ in this case?
(c) From my understanding, I have to use Divergence Theorem here $$\iint_{V} F.\hat n\mathrm dS = \iiint_{V} \nabla.F \mathrm dV $$ $$\iint_{V} F.\hat n\mathrm dS = \iiint_{V} -3(x^2+y^2-2z) \mathrm dV $$ $$\iint_{V} F.\hat n\mathrm dS = -3\iiint_{V} (x^2+y^2-2z) \mathrm dV $$
Using Cylindrical coordinates $$\iint_{V} F.\hat n\mathrm dS = -3\iiint_{V} [(r^2\cos^2\theta+r^2\sin^2\theta-2z)] \mathrm rdzdrd\theta $$ $$\iint_{V} F.\hat n\mathrm dS = -3\int_0^{2\pi} \int_0^4 \int_0^5 [(r^2\cos^2\theta+r^2\sin^2\theta-2z)] \mathrm rdzdrd\theta $$ $$\iint_{V} F.\hat n\mathrm dS = 432\pi $$
Use the Divergence Theorem for the whole thing. This converts the surface integral to a volume integral:
$$ \iint_T F\cdot \hat{n} dS = \iiint_V (\nabla \cdot F) dV$$
This is much more doable to finish and it gets rid of $\hat{n}$.