We are asked to calculate as triple integral, through Gauss theorem, the surface integral of $z^2dxdy+x^2dydz+y^2dzdx$. We are given $D$ through the expression $0\leq x,y,z\leq 1$. I found that ${\rm div}(F)=0$, where $F=(z^2,x^2,y^2)$. Thus the triple integral through Gauss theorem is zero.
I would appreciate if you could guide me whether the specific way of thinking is correct. Thank you very much in advance.
No, the requested surface integral is not zero. Note that the surface integral of $z^2dxdy$ is $1$ on the top face and zero over the other $5$ faces of the cube $[0,1]^3$. By symmetry the surface integral of $z^2dxdy+x^2dydz+y^2dzdx$ is equal to $3$.
The same result can be obtained by applying Gauss' Divergence Theorem: let $F$ be a given smooth vector field, then $$\begin{align} \int_{[0,1]^3} {\rm div}(F)\,dV &= \int_S F \cdot n\,dS\\ &=\iint_{ [0,1]^2} F(x,y,0) (0,0,-1)\,dxdy+\iint_{ [0,1]^2} F(x,y,1) (0,0,1)\,dxdy\\ &+\iint_{ [0,1]^2} F(0,y,z) (-1,0,0)\,dydz+\iint_{ [0,1]^2} F(1,y,z) (1,0,0)\,dydz\\ &+\iint_{ [0,1]^2} F(x,0,z) (0,-1,0)\,dxdz+\iint_{ [0,1]^2} F(x,1,z) (0,1,0)\,dxdz.\end{align}$$ Now let $F=(x^2,y^2,z^2)$ and compare the two sides. Can you take it from here?