I'm asked to evaluate the flux $$\int_{S} rot K d\omega,$$ where $S$ is the region $-2 \leq z \leq 1, \sqrt{x^2+y^2}=1+z^2$ and $K$ is the vector field $(-zy,zx,z^2)$.
So when I evaluate directly the integral, I use as parametrization $$x=r\cos(\theta), y=r\sin(\theta), z=\sqrt{r-1}.$$ As rotation, I get $(-x,-y,2z)$.
As normal vector, I get $$(\cos(\theta),\sin(\theta),\frac{1}{2\sqrt{r-1}})X(-r\sin(\theta),r\cos(\theta),0)=(\frac{-r\cos(\theta)}{2\sqrt{r-1}},\frac{-r\sin(\theta)}{2\sqrt{r-1}},r)$$ Now, if I take the dot product of that with $(-r\cos(\theta), -r\sin(\theta),\sqrt{r-1})$, I get $$\frac{r^2}{2\sqrt{r-1}}+r\sqrt{r-1}$$ which seems really ugly, so I wanted to know if until now, everything I did is correct, and if yes, if I'm supposed to integrate this from $r=5$ to $r=2$ (given that $z=\sqrt{r-1}$) times $2\pi$.
Using the relation with Stoke's theorem, I get the integral over the border. As parametrization, I use $((1+z^2)\cos(\theta),(1+z^2)\sin(\theta),z)$. So the normal vector is $(2z\cos(\theta), 2z\sin(\theta),1)X(-(1+z^2)\sin(\theta), (1+z^2)\cos(\theta),0)=(-(1+z^2)\cos(\theta),-(1+z^2)\sin(\theta),2z(1+z^2))$.
The dot product of that with $(-z(1+z^2)\sin(\theta),z(1+z^2)\cos(\theta),z^2)$ is $z(1+z^2)^2\sin(\theta)\cos(\theta)-z(1+z^2)\cos(\theta)\sin(\theta)+2z^3(1+z^2)$. If we integrate that, we're just left with $2z^3(1+z^2)$. If we integrate that from $-2$ to $1$, we get $\frac{1}{4}+\frac{1}{6}-4-\frac{64}{6}$ times $4\pi$ which again seems a bit weird.
So I don't think that my results are correct and I would be happy if you could point out what I did wrong in both cases and how I should proceed instead. Thanks for your help !
Direct approach:
Can I suggest you use this parametrisation: $$ x = (1 + u^2) \cos \phi, \ \ y = (1 + u^2) \sin \phi, \ \ z = u \ \ \ \ \ (u \in [-2, 1], \ \phi \in [0, 2\pi])$$
[Notice how my parametrisation avoids square roots! The parametrisation that you chose, with $z = \pm \sqrt{1 - r}$, is problematic because you have to treat the separate cases where $z$ is positive or negative, giving you two integrals to evaluate instead of one. And besides, avoiding square roots makes the algebra nicer.]
As you say, the curl is
$$ \nabla \times \vec K=\begin{bmatrix} -x \\ -y \\2z \end{bmatrix}=\begin{bmatrix} -(1+u^2)\cos \phi \\ -(1+u^2)\sin\phi\\2u\end{bmatrix}$$ while is area element is $$ d \vec A= \frac{\partial\vec r}{\partial u} \times \frac{\partial \vec r}{\partial \phi}\ dud\phi = \begin{bmatrix} 2u\cos \phi \\ 2u\sin\phi\\ 1\end{bmatrix} \times \begin{bmatrix}-(1+u^2)\sin\phi \\ (1+u^2)\cos\phi \\ 0\end{bmatrix} dud\phi= \begin{bmatrix} -(1+u^2)\cos\phi \\ -(1+u^2)\sin\phi \\ 2u(1+u^2)\end{bmatrix}dud\phi$$ so the integral is $$ \iint (\nabla \times \vec K).d\vec A=\int_{u=-2}^{u=1}\int_{\phi=0}^{\phi=2\pi}\left( 1 + 6u^2 + 5u^4\right) = 108\pi $$
Using Stoke's theorem:
The region has two boundary components: $$ C_1: \ \ \ \ \ x = 2\cos \phi, \ y = 2\sin\phi, \ z = +1 \ \ \ \ \ (\phi \in [0, 2\pi])$$ $$ C_2: \ \ \ \ \ x = 5\cos \phi, \ y = 5\sin\phi, \ z = -2 \ \ \ \ \ (\phi \in [0, 2\pi])$$
[Notice how these boundary curves are parameterised by a single parameter $\phi$, and notice how the parameter $u$ that we used in the direct approach is now treated as a constant. (It is equal to $1$ on $C_1$ and $-2$ on $C_2$).]
For the orientations to be consistent, $C_1$ must be traversed anticlockwise, and $C_2$ clockwise. So \begin{align} \iint (\nabla \times \vec K) . d\vec A &= \oint_{C_1} \vec K . d\vec l - \oint_{C_2} \vec K . d\vec l \\ &= \oint_{\phi = 0}^{\phi = 2\pi} \begin{bmatrix}-2\sin\phi \\ 2\cos \phi \\ 1 \end{bmatrix} . \begin{bmatrix}-2\sin\phi \\ 2 \cos \phi \\ 0 \end{bmatrix}d\phi - \oint_{\phi = 0}^{\phi = 2\pi} \begin{bmatrix}10\sin\phi \\ -10\cos \phi \\ 4 \end{bmatrix} . \begin{bmatrix}-5\sin\phi \\ 5 \cos \phi \\ 0 \end{bmatrix}d\phi \\ &= 108\pi\end{align}
[Notice that the line elements $d\vec l = \frac{\partial \vec r}{\partial \phi}$ are tangential to the boundary curves. They are not normals!]