In Modern Robotics, there is a line that says, because we have 3 unitary constraints, and 3 orthogonality constraints for rotation matrices, summarized by the equation $R^T R = I$, and $R$ is 9-dimensional, that means $R$ has 3 free variables.
My understanding of free variables comes from linear algebra, where the number of "free variables" is essentially the dimensionality of the null space.
In order to argue that the previous setup has a 3-dimensional nullspace, the 3 unitary and 3 orthogonality constraints must be linearly independent constraints.
However, looking at these constraints:
These constraints are quadratic in terms of the rotation matrix variables. Therefore, linear independence does not make sense as a concept here.
How do I rigorously argue that $R$ has 3 "free" variables in this setting? How do you define "free" variables in a general way with non-linear constraint equations, instead of using the nullspace, which is a concept for linear constraint equations?
The appropriate "rigorous" notion to apply here is that $O(3)$, the set of rotations in three dimensions, is a three-dimensional Lie Group, which in particular means that it is a three-dimensional Smooth Manifold. This means essentially that $O(3)$ is "locally" described by three smooth coordinate functions, but this takes a bit of work to make precise.
The standard way to show that $O(3)$ is indeed a manifold is through the preimage theorem, which states that, given a smooth map $f:M\to N$ between manifolds, and $x\in M$ is a regular value, then $f^{-1}(\{x\})$ is a smooth manifold of dimension $\dim f^{-1}(\{x\})=\dim M-\dim N$ (and in fact a smooth submanifold of $M$).
Here, we can identify the space of $3\times 3$ matrices with $\mathbb{R}^9$ and the set of symmetric $3\times 3$ matrices with $\mathbb{R}^6$, and consider the map $f:\mathbb{R}^9\to\mathbb{R}^6$ defined by $f(A)=A^TA$ under this identification, so that $O(3)$ is then the level set $f^{-1}(I_3)$. The level set theorem then states that $O(3)$ is a smooth manifold if $I_3$ is a regular value of $f$, i.e. if the Jacobian of $f$ is surjective on all of $O(3)$.