We are interested in the volume of the bounded region above $z = \sqrt{3x^2 + 3y^2}$ inside $x^2+y^2+z^2=9$. To Find the volume, we need to calculate this:
$$ \iint_{x^2+y^2=3} \sqrt{9-x^2-y^2} - 3\sqrt{x^2+y^2}dxdy. $$
Switching to polar coordinates yields: $$\int_{\theta=0}^{2\pi}\int_{r=0}^3 (\sqrt{9-r^2} - 3r)rdrd\theta=2\pi[-\frac{1}{3}(9-r^2)^{3/2}-r^3]_0^3 = -36\pi.$$
But I guess my answer doesn't make sense since a volume above the $z$ axis has to be positive. Also the solution manual says the answer is $9\pi (2-\sqrt{3})$ which makes more sense. (At least it's positive ;) )
Note that$$\sqrt{3x^2+3y^2}=\sqrt3\sqrt{x^2+y^2}.$$Furthermore, you have $\sqrt3r\leqslant\sqrt{9-r^2}$ when and only when $r\leqslant\frac32$. So, the volume is\begin{align}\int_0^{2\pi}\int_0^{3/2}\left(\sqrt{9-r^2}-\sqrt3r\right)r\,\mathrm dr\,\mathrm d\theta&=2\pi\left[-\frac{r^3}{\sqrt{3}}-\frac13\left(9-r^2\right)^{3/2}\right]_{r=0}^{r=3/2}\\&=9\left(2-\sqrt3\right)\pi.\end{align}