Let $q \colon \mathbb{R}^n \to \mathbb{R}$ be a quadratic form with associated matrix $A \in $ Sym$(n, \mathbb{R})$, meaning that $q(x)=x^TAx$ for all $x$. Let us first assume that $q$ is non-degenerate, i.e. $A$ is invertible. My first question is topological:
1) How does one show that $O(q):=\{h \in GL(n,\mathbb{R}) \mid h^TAh=A\}$ is compact if and only if $q$ is positive or negative definite?
My second question is of differential geometry:
2) I can show that $O(q)$ is a submanifold of $GL(n,\mathbb{R})$ of dimension $\frac{n(n-1)}{2}$, but what happens if $q$ is degenerate? Is $O(q)$ still a submanifold, and of what dimension?
Any hint would be appreciated.
Since $A$ is symmetric, there is an orthogonal matrix $D$ (i.e., $D^{-1} = D^T$) so that $A = D^T \Lambda D$, where $\Lambda$ is diagonal. Since
$$h^T Ah = A \Leftrightarrow (DhD^T)^T \Lambda (DhD^T) = \Lambda$$
Thus $$O(q) = D^T O_{\Lambda} D,$$
where $O_\Lambda = \{ h: h^T \Lambda h = \Lambda\}$. Thus to answer (1), (2), it suffices to assume that $A$ is diagonal.
By choosing a different $D$ is necessary, assume that the diagonal entries $\lambda_i$ of $\Lambda$ are ordered by it's sign. If some of them $\lambda_{k+1}, \cdots, \lambda_l$ are zero, then it is easy to see that $h$ must be of the form
$$ h = \begin{bmatrix} A & 0 & B \\ 0 & G &0 \\ C& 0& D\end{bmatrix},$$
where $G \in GL(n-k,\mathbb R)$. This shows already that $O(q)$ is noncompact.
Thus we now assume that $\lambda_i\neq 0$ for all $i$. Let $B,S$ be the diagonal matrix with diagonal entries $B_i = \sqrt{|\lambda_i|}$, $S_i = \text{sgn} (\lambda_i)$ respectively. Then $$\begin{split} &h^T \Lambda h = \Lambda \\ \Leftrightarrow\ &h^T BSB h = B SB \\ \Leftrightarrow\ &((B^{-1} h^T B) S (B h B^{-1}) = S \\ \Leftrightarrow \ & (BhB^{-1})^T S (BhB^{-1}) = S\end{split}$$
Thus again we reduce to check the case for $\Lambda = S$ (Note that we also found that if $A$ is positive or negative definite, then $O(q)$ is compact since $S = \pm I$ and so $O(q)$ is diffeomorphic to the orthogonal group. On the other hand, that $O(q)$ is noncompact otherwise can be checked as suggested by Mike in the comment).
Of course these objects are well known. They are the indefinite orthogonal group $O(p,q)$. To show that it is indeed smooth, one may argue as usual, using the mapping $ h\mapsto h^T Sh$ and show that the tangent map is surjective for all $h$ in the preimage of $S$. But the smoothness also follows from the fact that all closed subgroup of a Lie group is automatically smooth, so I will just skip the checking.
The dimension of $O_S$ can be found to be the same as the orthogonal group.
Thus to sum up, all $O(q)$ are smooth manifolds, and if $A$ has $a$ positive eigenvalues, $b$ zero eigenvalues and $c$ negative eigenvalues, then $O(q)$ is diffeomorphic to $O(a,c) \times GL(b,\mathbb R)$ and it's dimension is $$b^2 + \frac{1}{2} (a+c) (a+c-1).$$