Calculate $\iint_S \vec F \cdot \vec n dS$.
While $S$ is between $z=x^2+y^2$, $z=\sqrt{x^2+y^2}$.
$\vec F=(3x,4y,-z)$. (The normal to the outside of S)
Using gauss law: $$\iint_S \vec F \cdot \vec n dS=\iint_D(\int^{\sqrt{x^2+y^2}}_{x^2+y^2}6)=6\int^{2\pi}_{0}(\int^1_0(r-r^2)r dr)d\theta)=12\pi\frac{1}{12}=\pi.$$
Normal way: On the paraboloid "$S_1$" : $\vec n=(2x,2y,-1)$.
$$\iint_D(\vec F)\cdot(\vec n)dxdy = \iint_D6x^2+8y^2+(x^2+y^2)dxdy=\int_0^{2\pi}[\int_0^1(7r^2+2r^2sin^2(\theta)dr]d\theta=2\pi\frac{7}{4}=\frac{7\pi}{2}.$$
On the cone $S_2$ : $\vec n=(\frac{-x}{\sqrt{x^2+y^2}},\frac{-y}{\sqrt{x^2+y^2}},1)$.
$$\iint_D(\vec F)\cdot(\vec n)dxdy=\iint_D-[\frac{3x^2+4y^2}{\sqrt{x^2+y^2}}+\sqrt{x^2+y^2}]=\int_0^{2\pi}-[\int_0^1(\frac{3r^2+r^2sin^2\theta}{r}+r)rdr]d\theta=-\int_0^{2\pi}[1+\frac{1}{3}+sin^2\theta] d\theta =- \frac{8\pi}{3}.$$
Now adding them together: $\frac{5\pi}{6}$. and according to my first attempt it needs to be $\pi$.
Where's the hidden mistake? (I'm more sure of gauss law attempt since it was easier, but I just want to make sure I don't have any hidden bad understanding of solving integrals).
Any feedback is really appreciated, thanks in advance!
You have two mistakes -
First: while calculating flux through paraboloid surface, you forgot Jacobian $r$.
$ \displaystyle \iint_D [6x^2+8y^2+(x^2+y^2)] \ dx \ dy$
$ = \displaystyle \int_0^{2\pi} \int_0^1 (7r^2 + 2 r^2 \sin^2 \theta) \ r \ dr \ d\theta$
$ = \displaystyle \int_0^{2\pi} \left(\frac{7}{4} + \frac{1}{2} \sin^2 \theta\right) \ d\theta$
$ \displaystyle = \int_0^{2\pi} \left(\frac{7}{4} + \frac{1}{2} \cdot \frac{1-\cos2\theta}{2}\right) \ d\theta$
$ \displaystyle = \int_0^{2\pi} \left(\frac{7}{4} + \frac{1}{4} - \frac{\cos2\theta}{4}\right) \ d\theta$
Integral of $\cos2\theta$ over $(0, 2\pi)$ is zero. That leads to an answer of $4\pi$.
Second: while calculating flux through cone, you have a mistake in integral.
$ \displaystyle \iint_D -\left(\frac{3x^2+4y^2}{\sqrt{x^2+y^2}}+\sqrt{x^2+y^2}\right) \ dx \ dy$
$ = \displaystyle \int_0^{2\pi} \int_0^1 - (4r^2 + r^2 \sin^2 \theta) \ dr \ d\theta = - 3\pi$
$\left(\text {again use } \sin^2\theta = \dfrac{1 - \cos2\theta}{2}\right)$
So the net flux through the closed surface is $\pi, \ $ which matches the answer you obtained by applying divergence theorem.