I'm having some problems trying to solve this type of problems. I'm asked to find the volume between the cone $ y^2\sin^2{a} = (x^2+z^2)\cos^2{a} $ and the sphere $x^2+y^2+z^2=b^2$ with $a,b\in\mathbb{R}$.
I suppose that I will have to use spherical coordinates (for example, $\rho=b$ substituting on the sphere equation) and find out the limits of integration considering that $dV = \rho^2 \sin \phi d\phi d\theta d\rho$ but I don't know how to procede. Any help is appreciated.
I like to set-up in Cartesian, before considering a conversion to polar or spherical.
Substitute $y^2 =\tan^2 a(x^2 + z^2)$ into the equation of the sphere to find the intersection of the two surfaces.
$\sec^2 a(x^2 + z^2) = b^2$
$2\int_{-b\cos a}^{b\cos a}\int_{-\cos a \sqrt{b^2-z^2}}^{\cos a\sqrt{b^2-z^2}}\int_{\tan a\sqrt{z^2-x^2}}^{\sqrt{b^2-z^2-x^2}} \ dy\ dx\ dz$
Okay, this doesn't look so bad. We can keep going in Cartesian, or we can convert to polar or spherical.
Polar
$x = r\cos \theta\\z=r\sin\theta\\y=y$
$y^2 = r^2\tan^2 a\\ y^2 = b^2 - r^2\\ r = b\cos a$
$2\int_0^{2\pi}\int_0^{b\cos a}\int_{r\tan a}^{\sqrt{b^2-r^2}} r\ dy\ dr\ d\theta$
Spherical
$x = \rho\cos \theta\sin\phi\\ z=\rho\sin\theta\sin\phi\\ y=\rho\cos\phi$
$\rho^2\cos^2\phi\cos^2 a = \rho^2\sin^2\phi\sin^2 a\\ \tan\phi = \cot a$
$2\int_0^{2\pi}\int_0^{\frac {\pi}{2}-a}\int_0^b \rho^2\sin\phi \ d\rho\ d\phi\ d\theta$
or
$x = \rho\cos \theta\cos\phi\\ z=\rho\sin\theta\cos\phi\\ y=\rho\sin\phi$
$2\int_0^{2\pi}\int_a^{\frac {\pi}{2}}\int_0^b \rho^2\cos\phi \ d\rho\ d\phi\ d\theta$
Update -- does the integral via polar equal the integral via spherical?
$2\int_0^{2\pi}\int_0^{b\cos a}\int_{r\tan a}^{\sqrt{b^2-r^2}} r\ dy\ dr\ d\theta\\ 2\int_0^{2\pi}\int_0^{b\cos a} r\sqrt{b^2-r^2} - r^2\tan a\ dr\ d\theta\\ 2\int_0^{2\pi} -\frac 13 (b^2-r^2)^\frac 23 - \frac 13 r^3\tan a |_0^{b\cos a} d\theta\\ \frac 23 \int_0^{2\pi} b^3 - b^3\sin^3 a - b^3 \cos^2a\sin a d\theta\\ \frac {4\pi}{3}b^3(1-\sin a)$
$2\int_0^{2\pi}\int_a^{\frac {\pi}{2}}\int_0^b \rho^2\cos\phi \ d\rho\ d\phi\ d\theta\\ \frac {2}{3}\int_0^{2\pi}\int_a^{\frac {\pi}{2}} b^3\cos\phi \ d\phi\ d\theta$ \frac {2}{3}\int_0^{2\pi} b^3(1-\sin a) \ d\theta$ \frac {4\pi}{3}b^3(1-\sin a)$