Let $F: \mathbb{R}^3 \to \mathbb{R}^3, F(x,y,z) = (xy,-x^2,x+z)$ be a vector field. Compute the surface integral over the set $S$ which is bounded by the plane $2x+2y+z=6$ in the set $\{(x,y,z) \in \mathbb{R}^3 : x,y,z\geqslant 0 \}.$
So I would need to calculate $$\iint_{T}f(\varphi(x,y)) \|\frac{\partial\varphi}{\partial x} \times \frac{\partial\varphi}{\partial y} \| \ dx \ dy$$
Parameterizing $F$ as $\varphi(x,y) = (x,y, 6-2x-2y)$ I have that $$\frac{\partial\varphi}{\partial x}=(1,0,-2) \text{ and } \frac{\partial\varphi}{\partial y}=(0,1,-2)$$
so $\|\frac{\partial\varphi}{\partial x} \times \frac{\partial\varphi}{\partial y} \| = \| (2,2,1)\| = 3.$
Computing the composition $f\circ\varphi$ I get that $f(x,y,6-2x-2y)=(x^2y,-y^2,x+6-2x-2y)$, thus the integral I would need to compute is $$\iint_{T} (x^2y,-y^2,x+6-2x-2y)\cdot 3 \ dx \ dy$$
but this seems wrong, this is just $f(\varphi(x,y)$ scaled by $3$. What am I missing here? Any hints would be appreciated.
I think that you should evaluate the flux of the vector field $\vec{F}$ through the given surface $S$, a triangle where the normal vector $\vec{n}$ is constant: $$\begin{align}\iint_S \vec{F} \cdot \vec{n} \ dS&=\iint_{T}\vec{F}(\varphi(x,y)) \cdot \left(\frac{\partial\varphi}{\partial x} \times \frac{\partial\varphi}{\partial y}\right) \ dx dy \\&=\iint_T (xy,-x^2,x+(6-2x-2y))\cdot(2,2,1)\, dx\,dy \end{align}$$ where $\varphi(x,y)=(x,y, 6-2x-2y)$ and $T=\{(x,y): x+y\leq 3,x\ge 0,y\ge 0\}$.
As a reference, see wikipage.