My course problem booklet (Mathematics BSc, second-year module in Algebra, unpublished) has a question,
For any $\textbf{x}, \textbf{y} \in\mathbb{R}^n,$ let $\textbf{x} \cdot \textbf{y} := \textbf{x}^T\textbf{y}$ (ignoring the brackets on the 1-by-1 matrix, so $\textbf{x} \cdot \textbf{y}\in\mathbb{R}$.) Prove that if $A\in O_n$ then $(A\textbf{x})\cdot(A\textbf{y})=\textbf{x}\cdot\textbf{y}$ for all $\textbf{x},\textbf{y}\in\mathbb{R}^n$.
The solution given in the solutions booklet is \begin{align} (A\textbf{x})\cdot(A\textbf{y}) & = (A\textbf{x})^T(A\textbf{y}) && (1) \\ & = (\textbf{x}^TA^T)(A\textbf{y}) && (2) \\ & = \textbf{x}^T(A^TA)\textbf{y} && (3) \\ & = \textbf{x}^TI\textbf{y} && (4) \\ & = \textbf{x}^T\textbf{y} && (5) \\ & = \textbf{x}\cdot\textbf{y}, \text{ as required.} && (6) \end{align} The step from line (2) to line (3) uses associativity. But what group is this property taken from? It seems to me that $\textbf{x}^TA^T$ and $A\textbf{y}$ are both in $\mathbb{R}^n$, while $A^TA$ is in $O_n$, so moving the brackets changes the groups to which the terms belong. Why is that OK?