Use the Divergence Theorem to find the outward flux of
$F = (6x^2 + 2xy)\vec{i} + (2y + x^{2}z)\vec{j} + (4x^{2}y^{3})\vec{k}$
across the boundary of the region cut from the first octant by cylinder $x^{2} + y^{2} = 4$ and the plane $z = 3$.
I have found $div F = 2(6x + y + 1)$ but I do not know how to set up the integral that can be evaluate after this step.
The divergence theorem states
$$ \iint_S \textbf{F}\cdot d\textbf{A} = \iiint_D (\nabla \cdot \textbf{F}) \,dV $$
where $D$ is the volume inside the cylinder $x^2 + y^2 = 4, x \ge 0, y \ge 0, 0 \le z \le 3$
All you have to do is integrate the divergence $\nabla \cdot \textbf{F}$ over this volume. Hope you know how to set up the bounds. The integral is:
$$ \int_0^3 \int_0^2 \int_0^\sqrt{4 - x^2} (12x + 2y + 2) \,dy \,dx \,dz$$
It might be easier to switch to cylindrical coordinates $(r, \theta, z)$
$$ \int_0^3 \int_0^{\pi/2} \int_0^2 (12r\cos\theta + 2r\sin\theta + 2) \,r \,dr \,d\theta \,dz$$