I'm really struggling to understand Stokes' Theorem. I tried this exercise:
Let D be the portion of $z=1-x^2-y^2$ above the xy-plane, oriented up, and let $\vec{F}=\langle xy^2,-x^2y,xyz\rangle$. Compute $$\iint_{D}^{}(\nabla\times \vec{F})\cdot \hat{n}dS$$ Here is my work: $$\nabla\times \vec{F}=\langle xz,-yz,-4xy\rangle$$
$$\vec{f}(r,\theta) = \bigl\langle r\cos\theta ,r\sin\theta ,1-r^2 \bigr\rangle$$
$$\frac{\partial\vec{f} }{\partial r}= \langle\cos\theta,\sin\theta,-2r\rangle$$
$$\frac{\partial \vec{f}}{\partial\theta }= \langle -r\sin\theta ,r\cos\theta ,0 \rangle$$
$$\frac{\partial\vec{f} }{\partial r}\times \frac{\partial \vec{f}}{\partial \theta}=\left \langle 2r^2\cos\theta ,2r^2\sin\theta ,r\right \rangle$$
$$\left \|\frac{\partial\vec{f} }{\partial r}\times \frac{\partial \vec{f}}{\partial \theta } \right \|=r\sqrt{3}$$ $$\widehat{n}= \biggl\langle \frac{2r\cos\theta}{\sqrt{3}} ,\frac{2r\sin\theta }{\sqrt{3}},\frac{1}{\sqrt{3}}\biggr\rangle$$
Integrating, I have $$\frac{1}{\sqrt{3}}\int_{0}^{2\pi }\int_{0}^{1}2r^2\cos^2\theta (1-r^2)-2r^2\sin^2\theta (1-r\cos\theta )-4r\sin\theta\cos\theta\,dr\,d\theta $$
$$=\frac{1}{\sqrt{3}}\int_{0}^{2\pi }\int_{0}^{1}-2r^2\sin^2\theta (1-r\cos\theta )+2r^2\cos\theta\, \theta (1-r^2)-2r\sin(2\theta )\,dr\,d\theta$$ After splitting the integral into three integrals, I have $$\frac{1}{\sqrt{3}}\int_{0}^{2\pi }\int_{0}^{1}-2r^2\sin^2\theta (1-r\cos\theta )\,dr\,d\theta=-\frac{2\pi }{3\sqrt{3}}$$
$$\frac{1}{\sqrt{3}}\int_{0}^{2\pi }\int_{0}^{1}2r^2\cos\theta (1-r^2)\,dr\,d\theta =0$$
$$\frac{1}{\sqrt{3}}\int_{0}^{2\pi }\int_{0}^{1}-2r\sin(2\theta )\,dr\,d\theta =0$$
$$=-\frac{2\pi }{3\sqrt{3}}+0+0$$
But the answer is zero. What am I doing wrong?
If you don't apply Stokes' theorem (that is, you insist on computing the integral of the curl over the surface $D$), then you would have
$$\begin{align} \iint_D(\nabla\times\vec F)\cdot\mathrm d\vec S&=\iint_D\langle xz,-yz,-4xy\rangle\cdot\vec n\,\mathrm dS\\[1ex] &=\iint_D \langle r(1-r^2)\cos\theta,-r(1-r^2)\sin\theta,-4r^2\cos\theta\sin\theta\rangle\cdot\vec n\,\mathrm dS \end{align}$$
where all I've done here is compute the curl of $\vec F$ and composed it with $\vec f$ to replace $x\to r\cos\theta$, $y\to r\sin\theta$, and $z\to1-r^2$. The normal vector is
$$\vec n=\frac{\partial\vec f}{\partial r}\times\frac{\partial\vec f}{\partial\theta}=\langle2r^2\cos\theta,2r^2\sin\theta,r\rangle$$
So the surface integral reduces to
$$2\int_0^{2\pi}\int_0^1 \left(r^3(\cos(2\theta)-\sin(2\theta))-r^5\cos(2\theta)\right)\,\mathrm dr\,\mathrm d\theta$$
Note that you do more work than necessary, since
$$\vec n\,\mathrm dS=\left\|\frac{\partial\vec f}{\partial r}\times\frac{\partial\vec f}{\partial\theta}\right\|\frac{\left(\frac{\partial\vec f}{\partial r}\times\frac{\partial\vec f}{\partial\theta}\right)}{\left\|\frac{\partial\vec f}{\partial r}\times\frac{\partial\vec f}{\partial\theta}\right\|}\,\mathrm dr\,\mathrm d\theta=\left(\frac{\partial\vec f}{\partial r}\times\frac{\partial\vec f}{\partial\theta}\right)\,\mathrm dr\,\mathrm d\theta$$
so you don't strictly need to normalize the normal vector.
If you do wish to apply Stokes' theorem, the integral is trivial by comparison (I've omitted the details):
$$\int_C \vec F\cdot\mathrm d\vec r=-\int_0^{2\pi}\left(\cos^3\theta\sin\theta-\cos\theta\sin^3\theta\right)\,\mathrm d\theta$$
and both do indeed have the same value.