I was reading Moysés Nussenzveig's "Basic Physics Course 1" when I came across this excerpt in chapter 11, about rotations and angular momentum, in section 11.2, vector representation of rotations:
We could then think about associating a vector “θ” to a rotation through the angle θ, the direction of this vector being given by the direction of the axis. We have already seen, however (Fig. 3.12), that the quantity “θ” associated with a finite rotation, although having module, direction and sense, it would not be a vector, as the addition of quantities of this type is not commutative (cf. (3.2.5)). However, if instead of finite rotations we take rotations through infinitesimal angles δθ, we will now see that infinitesimal rotations are commutative and have a vector character. To do this, we will associate a vector with an infinitesimal rotation by the same procedure defined in Sec. 3.2 for finite rotations.
I actually understand that rotations don't commute, as they can be represented by matrices, and those don't commute. However, what allows, mathematically, that "infinitesimal rotations" can commute, while rotations in themselves, cannot? In what other mathematical situations do similar situations occur?
Rotations in $\Bbb{R}^3$ are element of the Lie group $SO(3)$. Ultimately, it is possible to write a rotation $R_\theta$ as $R_\theta = e^{\theta L}$, where $L$ is the generator of that rotation of angle $\theta$. As a side note, $L$ is an element of the Lie algebra associated to $SO(3)$ and can be interpreted physically as an angular momentum operator.
Now, let's consider two of these rotations, namely $R_{\theta_1}$ and $R_{\theta_2}$, with $\theta_1$ and $\theta_2$ being infinitesimal angles, hence $R_\theta = 1 + \theta L + \mathcal{O}(\theta^2)$ by Taylor expansion. In consequence, one has : $$ R_{\theta_1}R_{\theta_2} = \left(1 + \theta_1L_1 + \mathcal{O}(\theta_1^2)\right)\left(1 + \theta_2L_2 + \mathcal{O}(\theta_2^2)\right) = 1 + \theta_1L_1 + \theta_2L_2 + \mathcal{O}(\theta^2) $$ and similarly $$ R_{\theta_2}R_{\theta_1} = \left(1 + \theta_2L_2 + \mathcal{O}(\theta_2^2)\right)\left(1 + \theta_1L_1 + \mathcal{O}(\theta_1^2)\right) = 1 + \theta_1L_1 + \theta_2L_2 + \mathcal{O}(\theta^2), $$ hence $R_{\theta_1}R_{\theta_2} \approx R_{\theta_2}R_{\theta_1}$.
Alternatively, we could have tried to show that $R_{\theta_1}R_{\theta_2}R_{\theta_1}^{-1} = e^{\theta_1L_1}e^{\theta_2L_2}e^{-\theta_1L_1} = e^{\theta_2L_2 + \mathcal{O}(\theta^2)} \approx R_{\theta_2}$ with the help of the BCH formula. This result is also valid for other Lie groups and one-parameter groups in general. In the case of $SO(3)$, you might work with quaternions in order to derive the same proof.
Now, coming back to the interpretation part, an infinitesimal rotation is not really different from a (lightly curved) small translation, since an infinitesimal arclength is almost linear, hence its vectorial nature, as mentioned in your textbook. Mathematically, it can be seen through the expansion $R_\theta \approx 1 + \theta L$, where $L$ turns out to be interpreted as the angular momentum, which is a (pseudo-)vector.