If I have to calculate the surface integral of $\iint_S A \cdot n\ \mathrm {ds}$ where $A= 3zi-2xj+5x^2zk$ and $S$ is the surface of the cylinder $x^2+y^2=4$ and lying between $z=0$ and $z=4$ in the first octant.
so $\nabla \cdot A= 5x^2$ then can I simply use the divergence theory and say
$$\iint_S A \cdot n\ \mathrm {ds} = \int_0^4 \int_{-2}^2 \int_{-2}^2 5x^2 \mathrm {dx dy dz}$$
because $x^2+y^2=4$ is a circle with a radius of $2$ therefore $x$,$y$ lie between $-2$ and $2$?
also is there an order for integration?
You have chosen Cartesian coordinates to work in, which is fine, but your integration limits, as they stand, do not represent the portion of a cylinder as described for the problem.
Since the cross-sections of the cylinder parallel to the $ \ xy-$ plane are circles described by $ \ x^2 \ + \ y^2 \ = \ 4 \ $ , if you make the calculation in Cartesian coordinates, you will need to express one coordinate, say, $ \ y \ $ , through the functions $ \ y \ = \ \pm \ \sqrt{4 \ - \ x^2} \ $ . As we only need to work with the projection of these cross-sections onto the first quadrant of the $ \ xy-$ plane, we only require the "positive semi-circle" function, and the integration in $ \ x \ $ will be over the interval $ \ 0 \ \le \ x \ \le \ 2 \ $ . Applying the Divergence Theorem here will make the surface integral equal to
$$ \iint_S \ \mathbf{A} \cdot \mathbf{\hat{n}} \ \ dS \ \ = \ \iiint_V \ \nabla \cdot \mathbf{A} \ \ dV \ \ = \ \ \int_0^4 \int_0^2 \int_0^{\sqrt{4 \ - \ x^2}} \ 5x^2 \ \ dy \ dx \ dz \ \ . $$
[If you prefer the order of integration $ \ dx \ dy \ dz \ $ , the integral would be $ \ \int_0^4 \int_0^2 \int_0^{\sqrt{4 \ - \ y^2}} \ 5x^2 \ \ dx \ dy \ dz \ \ $ , covering the same projected quarter-circles. ]
$$ \ \ $$
Since the surface is cylindrical, a more convenient choice of coordinates may well be cylindrical. This requires only conversion of the integration over the horizontal cross-sections to polar coordinates. As we are working in the first quadrant, the integration in angle will "run" over $ \ 0 \ \le \ \theta \ \le \ \frac{\pi}{2} \ $ , and the integration in radius covers the interval $ \ 0 \ \le \ r \ \le \ 2 \ $ . With the conversion of the divergence function to polar coordinates, and the requisite set of "infinitesimal volume elements", the volume integration becomes
$$ \iiint_V \ \nabla \cdot \mathbf{A} \ \ dV \ \ = \ \ \int_0^4 \int_0^{\pi / 2} \int_0^2 \ 5 \ (r \ \cos \ \theta)^2 \ \ r \ dr \ d\theta \ dz \ \ . $$
Note that in this form, the triple integral is nicely separable.
$$ \ \ $$
As Ted Shifrin points out, however, we would still need to make some surface integral computations on the other "faces" of the "quarter-cylinder" that we've described. There are four of these in all, since we need to look at the quarter-circles at $ \ z \ = \ 0 \ $ and $ \ z \ = \ 4 \ $ , the rectangular face of height 4 and width 2 in the $ \ yz-$ plane, and the face of the same dimensions in the $ \ xz-$ plane. These are, respectively,
$$ \mathbf{z \ = \ 0 \ :} \quad \iint_{S_1} \ \mathbf{A} \cdot \ ( -\mathbf{k}) \ \ dS \ \ = \ 0 \ \ , $$
$$ \mathbf{z \ = \ 4 \ :} \quad \iint_{S_2} \ \mathbf{A} \cdot \ \mathbf{k} \ \ dS \ \ = \ \ \iint_{S_2} \ 5x^2 \ \cdot \ 4 \ \ dS \ \ , $$
over the quarter-circle of radius 2 in the first quadrant,
$$ \mathbf{x \ = \ 0 \ :} \quad \iint_{S_3} \ \mathbf{A} \cdot \ (-\mathbf{i}) \ \ dS \ \ = \ \ \int_0^4 \int_0^2 \ -3z \ \ \ dy \ dz \ \ , $$
and
$$ \mathbf{y \ = \ 0 \ :} \quad \iint_{S_4} \ \mathbf{A} \cdot \ (-\mathbf{j}) \ \ dS \ \ = \ \ \int_0^4 \int_0^2 \ 2x \ \ \ dx \ dz \ \ . $$
The non-zero fluxes through these latter three surfaces would then need to be added to our result from the volume integral above.
While none of the last three integrals is particularly difficult, the total amount of work required to apply the Divergence Theorem here is a bit greater than what is needed to carry out the single surface integral over the curved cylindrical face of the quarter-cylinder.