Given vector field $F=\langle xyz,x,e^{xy}\cos z\rangle$ and a semi-sphere $x^2+y^2+z^2\le 1$ over $z=0$ and a normal vector $n$ to the surface, calculate $\iint \operatorname{curl}F\cdot ndS$.
Because $$\operatorname{curl}F=\langle xe^{xy}\cos z,xy−ye^{xy}\cos z,1-xz\rangle$$ it looks like a nightmare to calculate $\iint \operatorname{curl}F\cdot ndS$ directly.
So I thought we can define a new field $G=\operatorname{curl}F$. Also let $S$ be the surface of semi-sphere and $B$ be the surface which closes the semi-sphere from the bottom.
Then $$\iint_S \operatorname{curl}F\cdot ndS=\iint_{S\cup B} \operatorname{curl}F\cdot ndS-\iint_B \operatorname{curl}F\cdot ndS\stackrel{\text{by Gauss theorem}}{=}\\ =\iiint_{S\cup B}\operatorname{div}(\operatorname{curl}F)-\iint_B \operatorname{curl}F\cdot ndS$$ Because $\operatorname{div}(\operatorname{curl}F)=0$ always then we just need to calculate the integral over $B$ and it's much easier because the normal unit vector is essentially the $z$ axis in the negative direction. $$ \iint_{B} \operatorname{curl}F\cdot ndS=\iint_{ B} \operatorname{curl}F\cdot \langle0,0,-1\rangle dS=\iint_{B} (xz-1) dS\\ \stackrel{\text{z=0}}{=}\iint_{B}-1 dS\stackrel{\text{circle area}}{=}-\pi $$ Thus: $$ \iint_S \operatorname{curl}F\cdot ndS=\pi. $$ Is this in the right direction? I'm particularly not sure if $\operatorname{div}(\operatorname{curl}F)=0$ in this case.
Assume ${\bf F}$ as given, and assume that the upper hemisphere $S: \>x^2+y^2+z^2=1,\ z\geq0$ is oriented upwards. The surface $S$ has a boundary cycle $\partial S$ which is the unit circle in the $(x,y)$-plane, oriented counterclockwise.
We are told to compute the flux integral $$\Phi:=\int_S{\rm curl}({\bf F})\cdot{\bf n}\>d\omega\ .\tag{1}$$ This computation can be performed in three ways:
(i) Compute ${\bf C}:={\rm curl}({\bf F})$ as a function of $x$, $y$, $z$, use the parametric representation $${\bf r}(\phi,\theta):=\bigl(\cos\theta\cos\phi,\cos\theta\sin\phi,\sin\theta\bigr)$$ ($\phi$ and $\theta$ are GPS coordinates) for $S$ and compute the surface integral as given: $$\Phi=\int_0^{\pi/2}\int_0^{2\pi}{\bf C}\bigl({\bf r}(\phi,\theta)\bigr)\cdot\bigl({\bf r}_\phi\times{\bf r}_\theta)\>d\phi\>d\theta\ .$$ (ii) Use Stokes' theorem to convert $(1)$ into a line integral along $\partial S$; then compute this line integral: $$\Phi=\int_{\partial S}{\bf F}\cdot d{\bf r}\ .\tag{2}$$ Going this way you don't even have to compute ${\bf C}$, but you need to parametrize $\partial S$: $${\bf r}(\phi)=(\cos\phi,\sin\phi,0)$$ and plug this into $(2)$.
(iii) You have chosen a third way, namely using Gauss' theorem. This theorem deals with a three-dimensional solid $B$ and its boundary surface $\partial B$. We define $B$ to be the half ball bounded by $S$ and the unit disc $U$ in the $(x,y)$-plane oriented downwards. Gauss' theorem then gives $$\int_{\partial B}{\bf C}\cdot{\bf n}\>d\omega=\int_B{\rm div}({\bf C})\>{\rm dvol}=0\ ,$$ since ${\rm div}\circ{\rm curl}=0$. From $\partial B=S+U$ it follows that $$\Phi=-\int_U {\bf C}\cdot{\bf n}\>{\rm d}(x,y)=\int_U C_3(x,y,0){\rm d}(x,y)\ .$$ It remains to correctly compute $C_3$, which I leave to you.