I want to refer to a mathematics book that explains the n-dimensional rotation matrix or rotation transformation.
Wikipedia concentrates most on 2D or 3D. There are things that one can say definition here and there, but I think it is not a good idea to use the definition there. Actually they don't seem to be definitions.
Strang's "Linear Algebra", Barret O'neill's "Elementary Differential Geometry" deal only with 2D or 3D cases. I think physicist are more interested in the general case, due to the theory of relativity. I found one explanation in "Geometrical Methods of Mathematical Physics" by Bernard Schutz. But I think it doesn't define the rotation matrix.
Artin's "Geometric Alegebra" defines the rotation group as an isometry $\sigma:V\to V$ such that $\det\sigma=1$. But the language there is so abstract that I can't catch any of them.
Can anyone give a reference that defines rotation transformation on $\mathbb R^n$ and state as a property that $A$ is a rotation matrix if and only if $A\in SO(n)$?
This is the end of the question and the below is what I wanted to do. I wanted to prove that if $A\in SO(n)$, then $A$ is a rotation about a line through the origin in $\mathbb R^n$. So I need to define the rotation transformation(or matrix) in $n$ dimensional Euclidean space.
The definition Artin gives is probably the most transparent and standard:
I think you should spend time trying to understand this rather than discarding it as "too abstract." It's really quite concrete.
To split hairs for a second, the most important nature of a rotation is that it is a transformation, not just a matrix. A matrix is just a particular way to represent a transformation. That's why the definition above emphasizes the qualities that make it a rotation: it does not mess with distances, and it does not mess with orientation. (Out of infinitely many choices of bases it can have infinitely many different matrix representations.)
That isn't even true in $\mathbb R^2$, as I'm sure you can see.
It happens to be true for rotations in $\mathbb R^n$ for odd $n$ due to the fundamental theorem of algebra, because it says each rotation (like in Artin's definition) has an eigenvector. But even then I doubt it's what you wanted: some rotations in $\mathbb R^n$ preserve much more than just one line through the origin.
And for even $n$'s, you still might not preserve any line: take, for example
$\begin{bmatrix}0&-1&0&0 \\ 1&0&0&0 \\ 0&0&0&-1\\ 0&0&1&0\end{bmatrix} $ as an $\mathbb R$ linear transformation.