Let $z=\sqrt{x^2+y^2}$, $z\leq 3$ and we wish to verify the divergence theorem for
a) $F={2x, 2y,1]$, b) $F={2x, 2y,2z]$ c) $F={2x, 2y,4z]$.
Volumintegral of these are
a) $36 \pi$ b) $54 \pi$ c) $72 \pi$.
For example: $\int_0^{2\pi} \int_0^3 \int_r^3 4 r dz dr d\theta = \int_0^{2\pi} \int_0^3 4r (3-r) dr d\theta = 2 \pi (6r^2-4/3 r^3)_0^3= 36 \pi$.
The otherside of divergence theorem for the top of the cone gives:
a) $9 \pi$ b) $54 \pi$ c) $108 \pi$
and the surface integral for the cone sides are:
a) $27 \pi$ b) $0$ c) $-36 \pi$
The unity normal vector used for the top is [0,0,1] and for side is $\frac{[rcos \theta, r\sin \theta, -r]}{r\sqrt{2}}$. Am I calculating correct?
No, this is not right. The cone has its apex at the origin and its base at $z=3$ (it is "standing on its tip") so the outward pointing normal points out radially in the $r$ direction and slightly down the $z$ axis. (You have the right normal for the base of the cone).
Also I don't know how you go that formula but the norm of the vector is $r\sqrt{2}\neq 1$. The correct outward-pointing cone normal is $$\frac{1}{\sqrt{2}} \left(\cos\theta, \sin\theta, -1\right).$$