A volume of a $3$-d cone with the apex at the origin of a Cartesian coordinate system $\mathbf{x} = [x, y, z]^T$ and with its axis of symmetry (given by a unit vector $\mathbf{e}_c$) aligned with the $z$-axis and an aperture angle of $2\alpha$ is given in the corresponding spherical coordinates by
$$ 0\le r \le a \\ 0 \le \theta \le 2\pi \\ 0 \le \phi \le \alpha $$
What does this definition become after we rotate the Cartesian coordinates by a matrix $\mathbf{R}$, i.e. $\mathbf{x'}$ = $\mathbf{Rx}$?
It seems to me that, in general, the new ranges of $[r', \theta', \phi']$ will become functions of other coordinates, e.g. $\phi_{Max}' = f(r',\theta')$, but how do we find these functions and the ranges? I guess we need to take projections of the cone onto the planes defined by new coordinates, but I am not clear on how exactly this is to be done and I am not sure if this will actually lead to a solution.
I thought I could start by first defining the ranges in the original $\mathbf{x}$ coordinates and then substituting the expression for $[x,y,z]$ in terms of the new $[x',y',z']$ coordinates, but if I take the equation of the cone in the original coordinates $\mathbf{x}$, which (if I am not mistaken) are given by the following implicit equations
$$ 0 \le z \le a \cos{\alpha} \\ x^2 + y^2 \le z^2 \tan^2{\alpha} $$
then I am not even able to define these as ranges for each of $[x, y, z]$. So I am stuck at the very beginning.
I am looking for an "algebraic" approach since I would need to extend this to multiple dimensions.
Add 1
I suppose one can also try substituting the inverse relationships between spherical and Cartesian coordinates, i.e. $ r = \sqrt{x^2+y^2+z^2}$ etc, into the first set of inequalities above and then substitute the expressions for $\mathbf{x}$ in terms of $\mathbf{x'}$, i.e. $x = \left( \mathbf{R}^{-1}\mathbf{x'} \right)_{1}$, but not sure if this would actually work and I suspect that there is a simpler method.
If I try to work through it, I get to
$$ 0\le \sqrt{x^2+y^2+z^2} \le a \\ 0 \le \mathrm{arctan2}(y,x) \le 2\pi \\ 0 \le \arctan\left( \frac{\sqrt{x^2+y^2}}{z} \right) \le \alpha $$
which then needs to be split into $3$ sets for simultaneous inequalities for the different values of $x$ in $\mathrm{arctan2}()$.
So, for $x > 0$ we have that $ -0.5\pi < \mathrm{arctan2}(y,x) < 0.5\pi$ and hence
$$ 0\le x^2+y^2+z^2 \le a^2 \\ -\infty < \frac{y}{x} < + \infty \\ 0 \le \frac{\sqrt{x^2+y^2}}{z} \le \tan(\alpha) $$
which kind of leads back to the same point as before. So, no progress with this approach, it seems.
Add 2
The first set of inequalities in my opening post above defines a volume of a "spherical cone" (which is a conical "wedge" of a sphere); a volume of a cone is instead defined by
$$ 0 \le \theta \le 2\pi \\ 0 \le \phi \le \alpha \\ 0\le r \le a \frac{\cos{\alpha}}{\cos{\phi}} $$
As suggested by @Circle Lover , we can calculate an angle between the rotated axis of the cone and a radial unit vector in spherical coordinates, which means
$$ 0 \le \arccos{\left( \mathbf{e}_r \cdot \mathbf{R}\mathbf{e}_c \right)} \le \alpha \\ 0 \le r \le a \frac{\cos{\alpha}}{\mathbf{e}_r \cdot \mathbf{R}\mathbf{e}_c} $$
but this is still missing one inequality to comlete the definition.
$r$ is untouched by the rotation, we can work with $r=1$.
After rotation, the cone axis will go in some direction $(u, v, w)$ (a unit vector).
If suffices to express that the cone aperture is $\alpha$ by means of a scalar product:
$$u\cos\theta\sin\phi+v\sin\theta\sin\phi+w\cos\phi=\cos\alpha.$$
This gives you a $\theta,\phi$ relation.