The equation for an open cylinder is $x^2+y^2=1$; where $0\leq z \leq 3$. The vector field is $\mathbf{F} = yx^2\mathbf{i}+z^3\mathbf{j}+y+z\mathbf{k}$.
How do you apply Stokes's theorem to find the flux?
Basically, I need help to compute $$\iint (∇×\mathbf{F})⋅\mathbf{dS} $$
I understand how to get $$(∇×\mathbf{F}) $$. But the issue for me is how to obtain $$dS$$
On
Your question is how to compute the normal unit vector of a cylinder. Well, an arbitrary vector pointing outward the cylinder is given, in cartesian coordinates, by $x\hat{x}+y\hat{y}$ for every $z$. If we want to turn it int a unit vector, we just have to divide it by its norm $(x\hat{x}+y\hat{y})/ \sqrt{x^{2}+y^{2}} $. But, because of the symmetry of your problem, you better change cartesian coordinates to cylindrical coordinates: $$x = r\cos\theta \quad y=r\sin\theta \quad z=z$$ Now, your 'infinitesimal element of area' ${\bf{dS}}$ is simply $\hat{n}dS$ where $\hat{n}$ is the unit normal pointing outward the cylinder. Thus, in cylindrical coordinates, we have: $$ {\bf{dS}} = \hat{n}dS = (\cos\theta\hat{x}+\sin\theta \hat{y})rd\theta dz$$
On
You could use the divergence theorem.
Over a closed surface $\iiint_V \nabla\cdot F\ dV = \iint_S F\ dS$
We can cap the ends of the cylinder to close the surface.
$\iiint_V 2xy + z\ dV = \iint_S F\ dS + \iint_{D_1} -y\ dD_1 + \iint_{D_2} y+3 dD_2$
$\frac {9\pi}{2} = \iint_S F\ dS + 3\pi\\ \frac {3\pi}{2}$
A good step to take first would be to convert from Cartesian to cylindrical coordinates. I'll use the $(r,\theta,z)$ convention. We know that $$ \begin{array}{l} x=r\cos( \theta )\\ y=r\sin( \theta )\\ z=z \end{array}$$ And our unit vector conversions, $$\begin{bmatrix} \hat{\mathbf{r}}\\ \hat{\boldsymbol{\theta }} \end{bmatrix} =\begin{bmatrix} \cos( \theta ) & \sin( \theta )\\ -\sin( \theta ) & \cos( \theta ) \end{bmatrix}\begin{bmatrix} \hat{\mathbf{i}}\\ \hat{\mathbf{j}} \end{bmatrix}$$ And of course the $z$s are identical. Inverting this matrix yields $$\begin{bmatrix} \hat{\mathbf{i}}\\ \hat{\mathbf{j}} \end{bmatrix} =\begin{bmatrix} \cos( \theta ) & -\sin( \theta )\\ \sin( \theta ) & \cos( \theta ) \end{bmatrix}\begin{bmatrix} \hat{\mathbf{r}}\\ \hat{\boldsymbol{\theta }} \end{bmatrix}$$ So we can rewrite $\mathbf{F}$ as $$\mathbf{F}( r,\theta ,z) =r\sin( \theta )( r\cos( \theta ))^{2}(\cos( \theta )\hat{\mathbf{r}} -\sin( \theta )\hat{\boldsymbol{\theta }}) +z^{3}(\sin( \theta )\hat{\mathbf{r}} +\cos( \theta )\hat{\boldsymbol{\theta }}) +z\hat{\mathbf{k}} .$$ And now a little algebra yields $$\mathbf{F}( r,\theta ,z) =\sin( \theta )\left( r^{3}\cos^{3}( \theta ) +z^{3}\right)\hat{\mathbf{r}} +\cos( \theta )\left( z^{3} -r^{3}\sin^{2}( \theta )\cos( \theta )\right)\hat{\boldsymbol{\theta}} +z\hat{\mathbf{k}} .$$ The $\mathbf{dS}$ is a surface element vector: $$\mathbf{dS}=\hat{\mathbf{n}} \boldsymbol{\cdot} \mathrm{d}A$$ Where $\hat{\mathbf{n}}$ is the unit normal vector to our surface. In this case this is easy to compute and is simply $$\hat{\mathbf{n}}=\hat{\mathbf{r}}$$ $\mathrm{d}A$ is our area differential, given by $$\frac{\mathrm{d}\theta\mathrm{d}z}{h_{\theta}h_z}$$ The $h$s are our scale factors, $$h_z=1 ; h_{\theta}=r$$ (If you're unfamiliar with this, see curvilinear coordinates).
Finally, the bounds of our integral are $r=1$, $0 \leq z \leq 3$, $0 \leq \theta \leq 2\pi$. I'll leave it to you to compute $\nabla \times \mathbf{F}$ and finish up the problem. Please tell me if I've made any errors!