We often consider $SO(n)$ as the group of rotations in $\mathbb{R}^n$ in the sense that the usual action of $SO(n)$ on $\mathbb{R}^n$ by matrix multiplication can be interpreted as a rotation operation in $\mathbb{R}^n$. Although I've always used this fact, I never had an intuitive understanding of why is that.
Basically, we know that the linear transformation associated to a matrix $A$ is inner product preserving if and only if $A\in O(n)$. In truth, if $A\in O(n)$ then we know that $AA^T = I$ and hence
$$\langle Ax, Ay\rangle=\langle x, A^TA y\rangle = \langle x,y\rangle , \quad \forall x,y\in \mathbb{R}^n.$$
Furthermore, if $A$ is inner product preserving, then $A\in O(n)$. Then we have two possible choices, $\det(A) = +1$ and so $A\in SO(n)$ and $\det(A) = -1$ and $A\in O(n)\setminus SO(n)$.
In that case, an element of $SO(n)$ is simply one inner product preserving linear transformation, that is, which doesn't change angles nor lengths, and which has determinant $+1$.
Now, from this rigorous definition of the elements of $SO(n)$ how can one see that the elements of $SO(n)$ can be though of as rotation operators? Why rotation is related to the elements of that group and how can one arrive at the definition of $SO(n)$ in trying to define properly the group of rotations?
$\newcommand{\Neg}{\phantom{-}}\newcommand{\Reals}{\mathbf{R}}\newcommand{\Basis}{\mathbf{e}}\newcommand{\Brak}[1]{\langle#1\rangle}$Perhaps the issue is to see that an orientation-preserving, distance-preserving transformation of $\Reals^{n}$ that fixes the origin $O$ (a common geometric definition of a "rotation about the origin") is precisely multiplication by an element of $SO(n)$. The following gives one formalization.
Theorem: Let $n \geq 1$ be an integer. If $T:\Reals^{n} \to \Reals^{n}$, then the following are equivalent.
$T$ is orientation-preserving, distance-preserving, and fixes the origin.
$T$ is an orientation-preserving, distance-preserving linear transformation.
There exists a matrix $A$ in $SO(n)$ such that $T(x) = Ax$ for all $x$ in $\Reals^{n}$.
Proof: (1. implies 2.) Consider first the plane case ($n = 2$). If $T_{0}$ is a distance-preserving plane transformation that fixes $O$, $\Basis_{1} = (1, 0)$, and $\Basis_{2} = (0, 1)$, then $T$ fixes the line $\ell_{1}$ through $O$ and $\Basis_{1}$ (because every point on $\ell_{1}$ is uniquely determined by its distances to $O$ and $\Basis_{1}$). Consequently, $T_{0}$ fixes every point of the plane, namely, is the identity transformation (since every point of the plane is uniquely determined by its distance to $\ell_{1}$ and its distance to $\Basis_{2}$).
Now assume $T$ is an orientation-preserving, distance-preserving plane transformation that fixes $0$. There exists a real number $\theta$ such that $$ \begin{aligned} T(\Basis_{1}) &= (\cos\theta, \sin\theta), \\ T(\Basis_{2}) &= \bigl(\cos(\theta + \tfrac{\pi}{2}), \sin(\theta + \tfrac{\pi}{2})\bigr) = (-\sin\theta, \cos\theta). \end{aligned} \tag{1} $$ Indeed, every point at unit distance from $O$ has the form $(\cos\phi, \sin\phi)$ for some real $\phi$, and if $T(\Basis_{1}) = (\cos\phi, \sin\phi)$, then the fact $T$ is distance-preserving implies $T(\Basis_{2}) = \pm(-\sin\phi, \cos\phi)$; choose $\theta = \phi$ or $\theta = \phi - \frac{\pi}{2}$ so that (1) holds.
If $R_{\theta}$ denotes rotation by $\theta$ about the origin, then $R_{\theta}^{-1} \circ T$ is distance-preserving and fixes $O$, $\Basis_{1}$, and $\Basis_{2}$, and is therefore the identity; this means $T = R_{\theta}$. This completes the proof of (1. implies 2.) for the case $n = 2$.
A similar argument works for $n > 2$; the fact that $T$ is orientation-preserving is required in order to pick a special orthogonal matrix $A$ so that $A^{-1} \circ T$ fixes the origin and all the standard basis vectors.
(2. implies 1. is obvious.)
(2. if and only if 3.) In the body of the post. (A linear transformation $T(x) = Ax$ preserves distances if and only if $\Brak{Ax, Ay} = \Brak{x, y}$ for all $x$ and $y$ in $\Reals^{n}$, if and only if $A^{T}A = I_{n}$, and is orientation-preserving if and only if $\det A = 1$.)
Remarks about rotations of $\Reals^{n}$ for $n \geq 4$.
A "fundamental rotation" of $\Reals^{n}$ (this term is not standard to my knowledge) fixes a linear subspace of dimension $(n - 2)$ and rotates the ($2$-dimensional) orthogonal complement just like an ordinary planar rotation. That is, the rotations most nearly generalizing one's planar and spatial intuition have an $(n - 2)$-dimensional "axis".
If $P_{1}$ and $P_{2}$ are oriented orthogonal planes in $\Reals^{4}$, and if $\alpha$ and $\theta$ are real, there is a rotation $T$ of $\Reals^{4}$ that rotates $P_{1}$ by $\theta$ and rotates $P_{2}$ by $\alpha\theta$. If $\alpha$ is irrational, the one-parameter subgroup of $SO(4)$ containing $T$, which in a suitable basis comprises the set of matrices $$ \begin{pmatrix} \cos\theta & -\sin\theta & 0 & 0 \\ \sin\theta & \Neg\cos\theta & 0 & 0 \\ 0 & 0 & \cos(\alpha\theta) & -\sin(\alpha\theta) \\ 0 & 0 & \sin(\alpha\theta) & \Neg\cos(\alpha\theta) \\ \end{pmatrix} $$ is an irrational winding on a torus; in particular, this one-parameter subgroup of rotations is not compact. Kinetically, if you set a pair of orthogonal planes in $\Reals^{4}$ turning at constant, incommensurable angular speeds, the planes never return to their starting configuration. Analogous phenomena occur in $\Reals^{n}$ with $n > 4$.
Euler's rotation theorem generalizes to odd-dimensional spaces: For every rotation $T$ of $\Reals^{2n+1}$, there exists a line through the origin that is fixed pointwise by $T$. The proof is the same as for the $3$-dimensional case: The characteristic polynomial of $A$ has odd degree, hence at least one real root, every real eigenvalue is either $+1$ or $-1$, and because $\det T = 1$, at least one eigenvalue is $+1$.