Calculate the flow of the vector field $F(x,y,z)=(xz,yz,-z^2)$ out of the surface $S=\{(x,y,z) \in \mathbb{R}^3:z^2=1+x^2+y^2, 2 \leq z \leq 3\}.$
If you calculate with divergence theorem or Stoke's theorem is zero, but by definition? I tried: take $\phi(x,y)=(x,y,\sqrt{1+x^2+y^2})$, so $F(\phi)=(x\sqrt{1+x^2+y^2},y\sqrt{1+x^2+y^2},-1-x^2-y^2)$ and $\dfrac{\partial \phi}{\partial x} \wedge \dfrac{\partial \phi}{\partial x} =\Big(-\dfrac{x}{\sqrt{1+x^2+y^2}},-\dfrac{y}{\sqrt{1+x^2+y^2}},1\Big).$ Then we have that, $$\int \int_{S}<F,n> = \int \int_{T} -1-2x^2-2y^2 = -60\pi,$$ where $T=\{(x,y)\in \mathbb{R}^2:3 \leq x^2+y^2 \leq 8\}.$ Where is my mistake? Could anyone help me?
It sounds like you misapplied the divergence theorem.
To apply the divergence theorem, write $$S_2 = \{(x,y,2) \in \mathbb{R}^3 \mid x^2 + y^2 < 3\} \\ S_3 = \{(x,y,3) \in \mathbb{R}^3 \mid x^2 + y^2 < 8\}$$ and note that $S \cup S_2 \cup S_3$ is the boundary of a bounded solid $V \subset \mathbb{R}^3,$ while $S, S_2, S_3$ are pairwise disjoint.
Applying divergence theorem, we have $$\iint_S\langle F, n\rangle d\sigma + \iint_{S_2} \langle F, -\vec{k}\rangle d\sigma + \iint_{S_3}\langle F, \vec{k}\rangle d\sigma = \iiint_V\text{div } F\, dV = 0$$
It follows that $$\begin{align*}\iint_S\langle F, n\rangle d\sigma &= -\iint_{S_2}\langle F, -\vec{k}\rangle d\sigma - \iint_{S_3}\langle F, \vec{k}\rangle d\sigma \\ &= -\iint_{S_2}2^2\,d\sigma - \iint_{S_3}-3^2 d\sigma \\ &= -2^2(3\pi) - (-3^2)(8\pi) \\ &= 60\pi\end{align*}$$
Note: The only difference between our answers is the sign, which just follows from different $n$ following from different interpretations of "out of the surface". You've use an "upward" $n$, while I've used a "downward" $n$ (since it is definitely "outward" with respect to $V$)