The group $O(n)$ is typically defined as the group of matrices $A \in M_{n\times n}(\mathbb R)$ such that $A A^t = 1$. If we have an $\mathbb R$-vector space $V$ with an inner product $\langle -,- \rangle $, we define $O_V$ as the group $O_V := \{\varphi \in Aut(V)|\langle \varphi u,\varphi v \rangle = \langle u,v \rangle, \forall u,v \in V\}$. We can identify $O(n)$ and $O_V$ after choosing an orthonormal basis. The same happens in the complex case, with Hermitian forms, when we have $U(n)$ and $U_V$.
But what happens with these groups if we change the signature of the form? We gain algebraically new groups? What happens with the matricial equation? Thanks in advance.
If our bilinear form has signature $n_+,n_-,n_0$, then we can identify it with the canoncial bilinear form $$ \langle e_i,e_j \rangle = \begin{cases} 1 & i=j, \quad i\leq n_+\\ -1 & i=j, \quad n_+ < i\leq n_+ + n_-\\ 0 & \text{otherwise} \end{cases} $$ In the case that $n_0 = 0$, this gives us precisely the indefinite orthogonal group (up to the selection of a suitable basis).