If for rotations $R_0$, $R_1$ we define the distance $d(R_0, R_1)$ to be the angle of $R_0 R_1^{-1}$ and given $r\in [0,\pi]$, what is the "volume" (normalised Haar measure) in SO(3) of the ball $\{R\in\mathrm{SO}(3):d(R, \mathrm{Id})<r \}$.
I want to know for putting prior probabilities on subsets of the space of rigid transforms in some software.
You easily can obtain the result from the fact that 2-dimensional measure of a radius-r sphere in SO(3), where r ≤ π, is proportional to $\sin^2\frac{r}{2}$. This follows from the fact that SO(3)’s universal cover, the Spin(3) group (also known as SU(2) or Sp(1)), is metrically a sphere S³ embedded into four-dimensional Euclidean space. You can find some details about that in Mean value of the rotation angle is 126.5° and in Wikipedia. To match given metric on SO(3) (namely, that the maximal distance is π, for 180° rotations), we should choose the radius for S³ to be 2, to obtain the length 2π for its “meridians” (from 1 to −1) and π for 180° rotations (from 1 to the “equator” of S³, composed of so named right versors). Then, the arc r on S³ subtends the angle r/2 (in radians), and spheres or radius r will be Euclidean 2-spheres of radius $2\sin\frac{r}{2}$.
We know that the SO(3) Haar (probability) measure for r = π must be 1 – the open ball of that radius excludes only a 2-dimensional submanifold of SO(3). Hence we haven’t care about constant factors (such as $2\sin\frac{r}{2}$ vs $\sin\frac{r}{2}$, or various π stuff) before computing $$ \int_0^r \sin^2\frac{s}{2}\,ds $$ and obtaining the normalizing factor for Haar measure from the r = π case.
Since $2\sin^2\varphi = 1 - \cos(2\varphi)$, we can rewrite the expression for volume, up to “1/2” factor, as: $$ \int_0^r (1 - \cos s)\,ds = r - \sin r\,.$$ We see, for r = π the formula gives π, and hence the normalized volume of the radius r ball is $$ \frac{r - \sin r}{π}.$$