To start, the question itself is:
- Evaluate the Surface integral $\iint_S \mathbf F \cdot d \mathbf S$
B) $\mathbf F(x, y,z) = <z,x-z,y>$, S is the triangle with vertices $(1,0,0),\,(0,1,0)\,(0,0,1)$, oriented downwards.
I know the formula I'm supposed to be using is $\iint_S \mathbf F \cdot d \mathbf S = \iint_D (-Pg_x-Qg_y+R)dA$, where S is given by $z=g(x,y)$, and by finding the normal vector and solving for z, I've gotten that $z=1-x-y$. From there, I found S, which is $<x,y,1-x-y>$, with the domains $0\leq x \leq 1,\, 0\leq y\leq 1$.
for $(-Pg_x-Qg_y+R)$, I got $g_x=-1,\,g_y=-1$ and I plugged into the formula
$(1-x-y)(-1)\,-\,\bigl(x-\left(1-x-y\right)\Bigr)(-1)\,+\,y$
I simplified this to $x+y$, and plugged it into the integral, which became:
$\int_0^1\int_0^1x+y\,dxdy$
My final answer was $1$, however according to the answer sheet it's supposed to be $-\frac13$.
My guess is I messed up when plugging into the formula, but I don't know how. Any help would be appreciated.
Your error is not in the values plugged in for the integrand, but in the bounds of the integral. Your surface is the plane $x+y+z=1$ for $x,y,z\geq0$. If we solve this equation at for $z$ and apply $z\geq0$, we get $1-x-y\geq0$, or $y\leq1-x$. Applying $0\leq y$ to this, we get $x\leq1$. Thus our integral should be:
$$\int_0^1\int_0^{1-x}x+y\ {\rm d}y\,{\rm d}x$$
Which evaluates to $\frac{1}{3}$.
Edit:
You said the answer is supposed to be $-\frac{1}{3}$, not $\frac{1}{3}$. So there is some error in the integrand, and I believe it has to do with the "oriented downward" text of the problem statement.
In particular, you used $dS=\langle g_x,g_y,1\rangle$ in your solution. I believe that the "downward-facing" part of the problem indicates that the $z$-component of this vector should be negative, so it should be negative what you found, that is, $dS=\langle -g_x,-g_y,-1\rangle$.
This results in an integrand of $-x-y$, and an answer of $\frac{-1}{3}$.