From Schaum's vector analysis:
My attempt:
$\vec n = \nabla S = 2x \hat i + 2z \hat k$
$ \hat n = \frac{1}{3} x \hat i + \frac{1}{3} z \hat k $
$ \vec A . \hat n = 2xz - \frac{xz}{3} = \frac {5}{3} xz$
$dS = \frac {dxdy}{ \hat n . \hat k}$ , $ \hat n . \hat k = \frac {z}{3}$
$dS = \frac {3}{z} dxdy$
$ \iint_S \vec A . \hat n dS = 5 \iint_R x dxdy$
I know $y$ ranges from $0$ to $8$ then
$5 \iint_R x dxdy = 40 \int x dx$
This is where I stop, I can't integrate $x$ from $0$ to $3$ directly and I can't substitute it with the equation $x^2 + z^2 = 9$ , how do I continue? Also without making use of the divergence theorem please.


The surface $S$ has 5 distinct parts, so
$$ \int_S {\bf A}\cdot {\rm d}^2{\bf S} = \sum_{k=1}^5\int_{S_k} {\bf A}\cdot {\rm d}^2{\bf S} $$
where
The surface differential for this case is ${\rm d}^2{\bf S} = (3\cos\theta \hat{x} + 3\cos\theta \hat{z}){\rm d}\theta {\rm d}y$, so the integral becomes
$$ \int_{S_1} {\bf A}\cdot {\rm d}^2{\bf S} = 45\int_0^{\pi/2}{\rm d}\theta\int_0^8{\rm d}y \sin\theta\cos\theta = 180 \tag{1} $$
In this case ${\rm d}^2{\bf S} = -\hat{y}r{\rm d}r{\rm d}\theta$ and
$$ \int_{S_2} {\bf A}\cdot {\rm d}^2{\bf S} = -2\int_0^3{\rm d}r\int_0^{\pi/2}{\rm d}\theta r^2\cos\theta = -18 \tag{2} $$
In this case ${\rm d}^2{\bf S} = +\hat{y}r{\rm d}r{\rm d}\theta$ and
$$ \int_{S_3} {\bf A}\cdot {\rm d}^2{\bf S} = \int_0^3{\rm d}r\int_0^{\pi/2}{\rm d}\theta r(8 + 2r\cos\theta) = 18(1 + \pi) \tag{3} $$
In this case ${\rm d}^2{\bf S} = -\hat{z}{\rm d}x{\rm d}y$ and
$$ \int_{S_4} {\bf A}\cdot {\rm d}^2{\bf S} = \int_0^3{\rm d}x\int_0^{8}{\rm d}y x = 36 \tag{4} $$
In this case ${\rm d}^2{\bf S} = -\hat{x}{\rm d}z{\rm d}y$ and
$$ \int_{S_5} {\bf A}\cdot {\rm d}^2{\bf S} = -\int_0^3{\rm d}z\int_0^{8}{\rm d}y 6x = -216 \tag{5} $$
The result is
$$ \int_S {\bf A}\cdot {\rm d}^2{\bf S} = \sum_{k=1}^5\int_{S_k} {\bf A}\cdot {\rm d}^2{\bf S} = 180 - 18 + 18(1 + \pi) + 36 - 216 = \color{red}{18\pi} $$