Find the work performed by the force field $${\bf F}(x, y, z) = x^2 {\bf \vec i} + 4xy^3 {\bf \vec j} + y^2 x{\bf \vec k}$$ on a particle that traverses the contour $C$; where $C$ is the boundary of the rectangular part of the plane $z = y$ above the rectangle $R=\{0\leq x\leq 1,0\leq y\leq 3\}$.
I have got to the point where $\nabla\times F = 2xy {\bf \vec i} - y^2 {\bf \vec j} + 4y^3 {\bf \vec k}$ and $n = {\bf \vec j} - {\bf \vec k}$
$$\iint (\nabla\times F) \cdot {\bf n} \;dS = -90$$
the second side of the solution I wasn't able to do.
You are on the right track. Just evaluate $$\iint_R \text{curl}({\bf F})\cdot {\bf n}\; dydx =-\int_{x=0}^1\int_{y=0}^3 (y^2+4y^3)\; dydx=-\left[\frac{y^3}{3}+y^4\right]_{0}^3=-(9+81)=-90.$$ The same result can be obtained directly by evaluating $$\int_{\partial R} {\bf F}\cdot d{\bf r}=\sum_{i=1}^4\int_{\gamma_i} {\bf F}\cdot d{\bf r}=0+\frac{1}{3}-90-\frac{1}{3}=-90$$ where the are four sides are:
$\gamma_1(t)=(0,t,t)$ for $t\in [0,3]$;
$\gamma_2(t)=(t,3,3)$ for $t\in [0,1]$;
$\gamma_3(t)=(1,3-t,3-t)$ for $t\in [0,3]$;
$\gamma_4(t)=(1-t,0,0)$ for $t\in [0,1]$.