A hand-wavy answer to the question in the title could be "3D rotations".
But I would like to build a solid and intuitive geometric understanding of the $SO(3)$ manifold together with the Lie group and the Lie algebra of 3D rotations.
I read this article on Lie algebras for 2D and 3D rotations, but I am confused when it comes 3D rotations. I would like to build a similar intuition for $SO(3)$ as I have formed for $SO(2)$.
2D Rotations
The article says the following about $SO(2)$:
$SO(2)$ is a 1D manifold living in a 2D Euclidean space, e.g. moving around a circle.
I seem to understand this intuitively; with 2D rotations, there is just 1 degree of freedom and in my mind the $SO(2)$ manifold looks exactly like the unit circle in $\mathbb{R}^2$.
Each point $p \in \mathbb{R}^2$ on the unit circle has its $X$ and $Y$ coordinate. The $X$ coordinate is the cosine and the $Y$ coordinate is the sine of the angle between the $X$ axis of the coordinate system and the vector $p$.
Given an arbitrary point $p$ on the unit circle, I can easily compute the angle $\alpha$ it corresponds to by e.g. $\alpha = cos^{-1}(p_x)$, where $p_x$ is the $X$ coordinate of the point $p$.
3D Rotations
This is where I am confused. The article says the following about $SO(3)$:
There are only 3 degrees of freedom in describing a rotation. But this object doesn’t live in 3D space. It is a 3D manifold, embedded in a 4-D Euclidean space.
If the $SO(3)$ manifold lives in 4D space, then each point on its surface seems to be an $r \in \mathbb{R}^4$. But what does each of the 4 components $r_x, r_y, r_z, r_w$ of $r$ represent (in the 2D case, it was sine and cosine of the angle)?
I also found this image. I assume that if the $SO(3)$ manifold is embedded in 4D space, then it cannot be visualized as a 3D sphere.
Moreover, the tangent space in the image is "just a 2D plane". If my understanding is correct, then the purpose of the tangent space (Lie algebra) is mainly to allow expressing angle parameters manipulation in a (linear) vector space (convenient for optimization problems) instead of a movement on the manifold (difficult for optimization problems).
Given that 3D rotations have 3 degrees of freedom, it would make more sense to me if the tangent space was 3D volume (with 3 DoF) instead of 2D plane (with 2 DoF). Is the image just an analogue for the $SO(3)$ manifold embedded in 4D?
I think I am missing something obvious. What is your intuitive understanding of the $SO(3)$ manifold and how do you visualize it?
Abstract group
Oftentimes, when one describes a group like $SO(3)$, they are referring to the abstract group, which is just a set of elements with a prescribed composition operation. An abstract group does not "live in" any larger space, and the "points" need not have any internal structure whatsoever. We can then endow this set with the structure of a smooth manifold and talk about Lie groups, Lie algebras, etc. This approach is theoretically useful, but when working with a specific group, it's often best to explicitly construct the group (or, more precisely, construct an isomorphic group) out of more familiar objects. There are many ways of doing this; I'll call these models of the group. Questions like "what are the points" or "what is the ambient space" don't make sense when applied to the abstract group, rather they are questions about a particular model of the group, and different models will yield different answers.
Defining Representation
The classical way of constructing a model of a group is to write it as a set of matrices (or, equivalently, as a set of linear maps $\mathbb{R}^n\to\mathbb{R}^n$; this is called a (linear) representation. $SO(3)$ is often defined this way, as the set of $3\times 3$ matrices $A$ satisfying $A^TA=I$ and $\det(A)>0$. In this model, we can view $SO(3)$ as a subset of $\mathbb{R}^9$, and it is in fact a smooth 3-dimensional submanifold, and the induced manifold structure on $SO(3)$ is exactly the Lie group structure. In this model, the points are $3\times 3$ matrices, and the ambient space is $\mathbb{R}^9$. If we define the Lie algebra $\mathfrak{so}(3)$ as the tangent space at the identity, we can identify it with a subspace of $\mathbb{R}^9$, and this subspace corresponds to the antisymmetric matrices, with the lie bracket given by the matrix commutator.
Quaternion Model
We can also construct a model of $SO(3)$ out of the quaternions $\mathbb{H}\cong\mathbb{R}^4$. The set of unit quaternions form a Lie group under quaternion multiplication, and this group is ismorphic to $SU(2)$, a double cover of $SO(3)$. Identifying each unit quaternion $q$ with its opposite $-q$, we obtain a group isomorphic to $SO(3)$. This allows one to localy view $SO(3)$ as equivalent to the 3-sphere in $\mathbb{R}^4$, but it is not actually an embedding onto the $3$-sphere, since there is the added step of identifying opposite points.
There are many other models of $SO(3)$, of course, but I find these two to be the most useful. When trying to build some intuition for an abstract group, it's useful to familiarize oneself with multiple ways of constructing it, and, perhaps more importantly, with the correspondences between these constructions.