when i solved by $ \iint F\cdot ndS $ ,i get an answer $a^2 h$ but on solving by $ \iiint \nabla\cdot F dxdydz $ ,i get $π a^2 h^2$. I would want to attach my work but I am not familiar with LaTeX and it would take me ages.
I would greatly appreciate if anyone guides me through the answer.
Your volume integral is ok.For surface integral consider the surface (i)$S_1(z=0)$ (ii)$S_2(z=h)$(iii)$S_3$ on curved surface.
$\iint s_1\,ds_1=\iint s_3\,ds_3=0$
For $S_2(z=h)$:$F(x,y,z)=y\hat{i}+x\hat{j}+h^2\hat{k}$ and $\hat n=\hat k$
$\iint F.\hat n\, ds_2=h^2\iint ds_2=h^2\pi a^2$