To preserve the inner product of vectors $x'\cdot x' = x\cdot x $
The linear transformation $ x' = U x $, needs to satisfy
$UU^T=I$.
How many free parameters are we left with?
And also, if we require that $\det(U) = +1$ shouldn't that reduce the number of free parameters by $1?$
We can count degrees of freedom and constraints. Since $U \in \mathbb R^{n\times n}$ then it has $n^2$ values (or degrees of freedom). However because it is orthogonal, there are several constraints. First $UU^T$ produces ones on the diagonal, which places a constraint on the length of each row of $U$. The number of length constraints is $n$. Second, because $UU^T$ produces zeros on the off-diagonals this means that the rows are all orthogonal to each other. This introduces another ${n \choose 2} = \frac{n(n-1)}{2}$ constraints. So the degrees of freedom minus the constraint yields:
Net DOF = $n^2 - n - \frac{n(n-1)}{2} = \frac{n(n-1)}{2}$