This was supposed to be a pretty straight forward question. The basic turn the crank and get the answer. I parametrized the integral:
$\vec r=(3cos(t),3\sin(t),z)$
$d\vec r=(-3sin(t),3\cos(t),0)d\theta+(0,0,1)dz$
$\hat n=(3cos(t),3sin(t),0)$ $dtdz$
and $F = (9cos(t),9sin(t),-2z^2)$
$\iint_{S} F.\hat n\mathrm dS$
$\iint_{S} (9cos(t),27sin^3(t),-2z^2).(3cos(t),3sin(t),0)dzdt$
$5\int_0^{2\pi}\int_0^5 [27cos^2(t)+81sin^4(t)] \mathrm dzdt$
$=(1755\pi)/4$
BUT! This is wrong. This ain`t the right answer.
I then tried the divergence theorem to convert it to a volume integral, and still, no luck.
And then, I tried breaking it into three parts. One over the curved surface of the cylinder, then the 2 flat (top-bottom) surfaces, and still failed to get the correct answer that seems to be: $-45\pi/4$.
Any idea why my approach to the problem is wrong?
Your calculation is correct, except for a couple of typos. You must be calculating the flux through the upper and lower surfaces incorrectly. On the upper surface, the unit normal is $(0,0,1)$ and $F(x,y,z)=(3x,y^3,-50)$ so the flux is $-50\cdot3^2\pi={-1800\pi\over4}.$ On the lower surface, the unit normal is $(0,0,-1)$ and the flux is $0$, since $z=0$.
Adding these to your result gives ${-45\pi\over4}$.