A fluid flow has flux density vector $F(x,y,z) = <x, -2x-y, z>$.Let $S$ denote the hemisphere $x^2 + y^2 + z^2 = 1, z \geq 0$, and let $n$ devote the unit normal that posts out of the sphere. Calculate the mass of fluid flowing through $S$ in unit time in the direction of $n$.
I first found the $\frac{\partial r}{\partial x} \times \frac{\partial r}{\partial y}$ and constructed the surface integral according to
$$\iint \vec{F} \cdot (\frac{\partial r}{\partial x} \times \frac{\partial r}{\partial y}) dV,$$ and then converted the variables into spherical coordinated, which gave me the integral
$$\int_0^{2π} \int_0^{π/2} [ \frac{\sin^3 \phi}{\cos \phi} (\cos2\theta - \sin2\theta) + \sin\phi \cos\phi ]d\phi d\theta$$
but in the first part of the integral while after integrating respect to $\phi$, I face with such an expression
$$(-\ln(\cos\phi) + \cos2\phi /2)|_0^{π/2},$$
which gives me $\infty$, so where is my mistake ?
Changing to spherical coordinates, we have $$\vec{r}=\sin\phi\cos\theta \vec{i}+\sin\phi\sin\theta\vec{j}+\cos\phi\vec{k}.$$ Then by some computation we can get $$\vec{r}_{\theta}\times\vec{r}_{\phi}=-\sin^2\phi\cos\theta\vec{i}-\sin^2\phi\sin\theta\vec{j}-\sin\phi\cos\phi\vec{k}.$$ So $$||\vec{r}_{\theta}\times\vec{r}_{\phi}||=\sin\phi.$$ Now $$\vec{F}\cdot\vec{n}=x\cdot \frac{x}{r}+(-2x-y)\cdot\frac{y}{r}+z\cdot\frac{z}{r}$$ where $r=1$. So $$\vec{F}\cdot\vec{n}=\sin^2\phi\cos 2\theta-\sin^2\phi\sin 2\theta+\cos^2\phi$$ Multiplying this by $\sin\phi$, you'll get the expression I posted in the comments.
Without using $\vec{n}$, notice that in the expression of $\vec{r}_{\theta}\times \vec{r}_{\phi}$, we need $z$-component to be positive. We have $$\int \vec{F}\cdot d\vec{S}=\int \vec{F}\cdot (\vec{r}_{\theta}\times \vec{r}_{\phi})\\ =\int (\sin\phi\cos\theta,-2\sin\phi\cos\theta-\sin\phi\sin\theta,\cos\phi)\cdot(\sin^2\phi\cos\theta,\sin^2\phi\sin\theta,\sin\phi\cos\phi)$$ This gives again the same expression as above.
Now using $z=g(x,y)=\sqrt{1-x^2-y^2}$ as you did, your expression of $\vec{r}_x\times\vec{r}_y$ is correct. But remember when doing this, you already project the surface onto $xy$-plane. So you are not going to spherical coordinates any more. Instead, you should use polar coordinates in $xy$-plane. So the denominator is $\sqrt{1-r^2}$, not $z=\cos \phi$. I believe it should lead to the same answer, although the computation might be more complicated due to the term $\sqrt{1-r^2}$.