Use a parametrization to find the flux $$\iint_S F \cdot n \, d\sigma$$ across the surface in a given direction: $$F=xy\overrightarrow i-z\overrightarrow k$$ outward (normal away from the z-axis) through the cone $$z=\sqrt{x^2+y^2} \qquad 0\le z\le 1 .$$
I managed to parametrize the cone using cylindrical coordinates:
$$r(r,\theta)=(r\cos\theta)\overrightarrow i+(r\sin\theta)\overrightarrow j+ r\overrightarrow k, \qquad 0\le r\le 1, 0\le \theta \le 2\pi .$$
However, the surface integral flux result I got was $-2\pi /3$ while the solution manual got $2\pi /3$ and while looking through the answer, I saw that this was due to the way we computed $r_{r}\times r_{\theta}$, I computed it as written while they did $r_{\theta}\times r_{r}$.
Since we can do the calculation both ways, does the sign matter in the result? If it does, how am I supposed to know which way to do it?
Yes, it matters, since the question specifies finding the "flux... in a given direction". Drawing a diagram and using the Right-Hand Rule (or just computing directly) shows that $\partial_r \times \partial_{\theta}$ points toward the $z$-axis at each (nonsingular) point, not away from it as specified.
This difference changes precisely the sign (in terms of the formula, this amounts to the anticommutativity of the cross product).