In 2 dimensional vector space, a rotation can be described as a rotation around an axis pointing along a third basis vector.
What would the equivalent operation look like in 3 dimensions? i.e. a "rotation" around an "axis" defined by a fourth basis vector?
Yes, in 3D (and only 3D) all rotations can still be described by an (oriented) axis $a$, and the angle of rotation $\theta$ (measured counterclockwise in a plane orthogonal to and oriented consistently with $a$, say) about that axis. This is the so-called axis-angle representation of a rotation.
The difference here is that the axis of rotation is just an ordinary vector in $\mathbb{R}^3$, unlike in 2D where the "axis of rotation" is a vector pointing out of the plane.