I have been reading into orthogonal groups and have some questions.
Recall the orthogonal group of matrices $O(n)=\{A\in \mathbb{R}^{n,n} : \det{A}\ne 0, A^T=A^{-1}\}$. This satisfies $\langle Ax,Ay\rangle=x^TA^TAy=x^Ty=\langle x,y\rangle$ for all $x,y\in \mathbb{R}^n$, where the inner product given is the dot product.
Suppose we have another inner product on $\mathbb{R}^n$, say $[-,-]$. One can similarly define the orthogonal group of this inner product, $O([-,-])=\{A\in \mathbb{R}^{n,n}: \det{A}\ne 0,\quad [Ax,Ay]=[x,y]\quad\forall x,y\in \mathbb{R}^n\}$.
1) From the calculation in paragraph 1, we have $O(n)\subset O(<-,->)$ i.e. the orthogonal matrices are a subset of the matrices invariant with respect to the dot product). Can we say that these two definitions are equivalent? From that same calculation in paragraph 1, I only see the requirement that $A^TA=I_n$ (i.e. $A^T$ is a left inverse). Is this the best we can conclude? (I am pretty sure not!)
2) More generally, can we relate the orthogonal matrices to the arbitrary inner product $[-,-]$? Are there any general conditions for which $O(n)\subset O([-,-])?$ (or even $O(n)=O([-,-])?$
Thoughts: It is well known that any inner product on $\mathbb{R}^n$ is of the form $[x,y]=x^TBy$, where $B$ is a symmetric positive definite matrix. Therefore, if $A\in O(n)$, then $[Ax,Ay]=x^TA^TBAy$, so I guess the requirement that $A\in O([-,-])$ is that $A^TBA=B$. That is, $A^{-1}BA=B$. From here I have no idea how to proceed further - this still does not feel very intuitive to me!
Edit: it seems $A^TBA=B$ characterises the matrices in $O([-,-])$ according to Orthogonal matrices only defined for standard inner product?.
Thanks!
The condition $A^TBA=B$ is equivalent to $(B^{-1/2}A^TB^{1/2})(B^{1/2}AB^{-1/2})=I$. Therefore $$ O([-,-])=\left\{B^{-1/2}QB^{1/2}:\ Q\in O(n)\right\}. $$ Now suppose $A\in O(n)\cap O([-,-])$. Then $AB=A(A^TBA)=(AA^T)BA=BA$. It follows that $O(n)\subseteq O([-,-])$ if and only if $B$ commutes with all real orthogonal matrices, but this occurs only when $B$ is a scalar matrix and in that case, $O(n)=O([-,-])$.