The question is: Find $\iint_Y F.N \ dS $ $$ F=(x^4+yz-x^5,5x^4y,z),\quad \text{The surface }Y=x^2+y^2-z^2=1, \ \ 0\leq z\leq 1 \quad N=\text{ the normal points away from z axis}$$
Here is how I thought:
I parameterized the surface with $r(x,y)=(x,y,\sqrt{x^2+y^2-1}) $ and i think the normal is: $(r'_x\times r'_y)= (\frac{x}{\sqrt{x^2+y^2-1}},\frac{y}{\sqrt{x^2+y^2-1}},-1)$ and then I calculated the integral in this way but the integral looks very complicated and awful, have I calculated correctly? and what is the limit of this integration? $$\iint \frac{x^5-x^6+5x^4y^2+x^2+y^2-1+\sqrt{x^2+y^2-1}xy}{\sqrt{x^2+y^2-1}}\ dx\,dy $$ Any suggestion would be great, thanks.
One of the ways would be to use divergence theorem and find the flux through entire closed surface which consists of the hyperboloid between $0 \leq z \leq 1, $ and circular top and bottom surfaces at $z = 0, z = 1$. We can then subtract the flux through circular surfaces at $z = 0$ and $z = 1$ which is an easier calculation (in fact you can see why the flux through the circular bottom at $z = 0 \ $ for the given vector field would be zero. So it is only the flux through the circular top that we may have to find and subtract).
$div\vec{F} = \nabla \cdot \vec{F} = 4x^3-5x^4 + 5x^4 + 1 = 1 + 4x^3$
Also as $x^3$ is an odd function and there is symmetry along $x-$plane ($YZ$), the integral simplifies further.
So, Flux $ = \displaystyle \int_V div\vec{F} \ dV = \int_0^{2\pi} \int_0^1 \int_0^\sqrt{1+z^2} \ r \ dr \ dz \ d\theta$
Now calculate the flux through the circular top and subtract from the above.
Also to comment on your working, the parametrization and your steps for double integral are correct. It is a tedious one! For limits,
From here on, you can see that for $0 \leq z \leq 1, 1 \leq x^2+y^2 \leq 2$. So you can convert it into polar coordinates as $x = r \cos\theta, y = r\sin\theta, z$.
$\int_0^{2\pi} \int_1^\sqrt2 \vec{F}\cdot (r_x \times r_y) \ r \ dr \ d\theta$
Another way to parametrize the hyperboloid would be,
$r(\theta, z) = (\sqrt{1+z^2} \cos \theta, \sqrt{1+z^2} \sin \theta, z)$ and
$(r'_\theta \times r'_z) = (\sqrt{1+z^2} \cos \theta, \sqrt{1+z^2} \sin \theta, z)$
But it does not make the integral any easier.