Evaluate $\displaystyle\iint \text{curl} (y\,\mathbf{i}+2\,\mathbf{j})\cdot n\; d\sigma$ where $\sigma$ is the surface in the first octant made up of the plane $2x+3y+4z=12$, and the triangles in the $(x,z)$ and the $(y,z)$ planes.
I cannot even out how to even attempt this problem. Any help in getting started is much appreciated.
Stokes' Theorem, for your case, says that $$\iint_S \nabla\times\mathbf{F}\;\cdot\;d\mathbf{S}=\oint_{\Gamma}\mathbf{F}\cdot d\mathbf{\Gamma}.$$ Now here $\Gamma$ is the boundary of the non-closed surface $S$. And you've got to use the right-hand rule to figure what's "outside" and what's "inside", and therefore which direction along the boundary of $S$ makes up $\Gamma$. Also, notational issue: $\text{curl}(\mathbf{F}):=\nabla\times\mathbf{F}.$ The $\cdot\; n \, d\sigma$ in your problem is equivalent to $\cdot \; d\mathbf{S}$ in Stokes' Theorem. So, if we figure out the path $\Gamma$ making up the boundary of your surface described by the plane and the two triangles, we should have $$\iint \text{curl} (y\,\mathbf{i}+2\,\mathbf{j})\cdot n\; d\sigma=\oint_{\Gamma}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\Gamma}.$$
To figure out $\Gamma$, you need to draw a picture. I think the triangles being talked about are the "projection" of the plane onto the axes mentioned. That is, the plane is a triangle in the first octant. To get the "triangle" in the $(x,z)$ plane, imagine a flashlight shining from somewhere on the $y$ axis, towards the origin, and the shadow it casts on the $(x,z)$ plane will be a triangle. I think that's the triangle they're talking about. If you draw the plane, you'll find it intersects the $x$ axis at $x=6$, the $y$ axis at $y=4$, and the $z$ axis at $z=3$. So the triangle on the $(x,z)$ axis would be comprised of the vertices $(0,0,0), (6,0,0),$ and $(0,0,3)$. Similarly, the triangle on the $(y,z)$ plane would be comprised of the points $(0,0,0), (0,4,0),$ and $(0,0,3)$.
So, if the surface is made up of the plane and these two triangles, that leaves a triangle in the $(x,y)$ plane as the boundary, and that triangle is comprised of (in the correct order of traversal as given by the right-hand rule) $(0,0,0), (6,0,0),$ and $(0,4,0)$. From $(0,0,0)$ to $(6,0,0)$ we'll call $\gamma_1;$ it has length $6$. From $(6,0,0)$ to $(0,4,0)$ we'll call $\gamma_2$. It has length $2\sqrt{13}$. Finally, from $(0,4,0)$ to $(0,0,0)$ we'll call $\gamma_3$, and it obviously has length $4$. So, we have $$\oint_{\Gamma}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\Gamma} =\int_{\gamma_1}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\gamma}+ \int_{\gamma_2}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\gamma}+ \int_{\gamma_3}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\gamma}.$$
$\gamma_1$ and $\gamma_3$ are the easiest: $$\int_{\gamma_1}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\gamma} =\int_0^6(y\,\mathbf{i}+2\,\mathbf{j}) \cdot (\mathbf{i} \, dx)=\int_0^6y \, dx=0,$$ since $y=0$ on $\gamma_1$.
For $\gamma_3$, we have $$\int_{\gamma_3}(y\,\mathbf{i}+2\,\mathbf{j})\cdot d\mathbf{\gamma} =\int_4^0(y\,\mathbf{i}+2\,\mathbf{j})\cdot(\mathbf{j} \, dy)=-2\int_0^4dy=-8.$$
For $\gamma_2$, we need a vector from the start point to the finish point, which would be given by $(0,4,0)-(6,0,0)=(-6,4,0)$. And here, in the grand tradition of leaving the harder parts to the student, I will let you finish.