The following is a problem from Munkres's Analysis in Manifolds.
Problem: Let $\mathcal O(3)$ denote the set of all orthogonal 3 by 3 matrices, considered as a subspace of $\mathbf R^9$.
(a) Define a $C^\infty$ function $f:\mathbf R^9\to\mathbf R^6$ such that $\mathcal O(3)$ is the solution set of the equation $f(\mathbf x)=\mathbf0$.
(b) Show that $\mathcal O(3)$ is a compact 3-manifold in $\mathbf R^9$ without boundary. [Hint: Show the rows of $Df(\mathbf x)$ are independent if $\mathbf x\in\mathcal O(3)$.]
In Euclidean space of course compactness is equivalent to being closed and bounded.
So far, I can only show that $\mathcal O(3)$ is closed: Considering $\mathcal O(3)$ as a subspace of $\mathbf R^9$, if we define a map $g:\mathbf R^9\to \mathbf R^9$ by $g(A)=A^TA$, then $g(A)=I$ if and only if $A$ is orthogonal. Hence, $\mathcal O(3)=g^{-1}(\{I\})$.Further, $g$ is polynomial in the entries of a matrix, so it is continuous. As $\{I\}$ is closed and $g$ is continuous, $\mathcal O(3)$ is closed.
But I'm a bit confused on how to show $\mathcal O(3)$ is bounded. I've seen other responses using the fact that $\|v\|=\|Av\|$ whenever $A$ is orthogonal; I don't understand why this shows that $\mathcal O(3)$ is bounded, though. There must be something conceptual that I'm missing here.
I'm also unable to come up with the function $f$ required in part (a). If I can find such an $f$, then I imagine it just takes a bit of linear algebra to show that $Df(\mathbf x)$ has linearly independent rows — in which case, the result for (b) follows.
Any hints on how to construct $f$ or show that $\mathcal O(3)$ is bounded would be greatly appreciated.
Let's use the operator norm on the space $M_{3}(\Bbb{R})$, which is defined as: for any $A \in M_3(\Bbb{R})$, \begin{align} \lVert A \rVert_{\text{op}}:= \sup\{\lVert Ax\rVert: \, \, \lVert x \rVert = 1\}, \end{align} where the norms $\lVert \cdot\rVert$ are the Euclidean norms on $\Bbb{R}^3$ (the one induced by the "standard" inner product). Then, for an orthogonal $A$, we have $\lVert Av \rVert = \lVert v\rVert$, so it follows that they have operator norm $1$: \begin{align} \lVert A \rVert_{\text{op}} &= 1. \end{align} Thus, $\mathcal{O}(3)$ lies on the unit sphere of $M_3(\Bbb{R})$ (relative to the operator norm). It is a relatively simple theorem to show that in finite-dimensions, every norm on a vector space generates the same topology. Thus, $\mathcal{O}(3)$, being a closed subset of a compact space (the unit sphere) is in fact compact.
Note that the map $g$ you defined can be written as follows: $g: M_3(\Bbb{R}) \to \text{Sym}_3(\Bbb{R})$, $g(A) := A^tA$. Then, $\mathcal{O}(3) = g^{-1}(\{I\})$. Here, $M_3(\Bbb{R})$ is a $9$-dimensional vector space, and $\text{Sym}_3(\Bbb{R})$ is a $6$-dimensional vector space. Now, you have to verify whether the hypothesis of the regular-value theorem is satisfied (this theorem is really a consequence of the inverse/implicit function theorem).
So, if you manage to show that for every $A \in \mathcal{O}(3) = g^{-1}(\{I\})$, the derivative $Dg_A: M_3(\Bbb{R}) \to \text{Sym}_3(\Bbb{R})$ is surjective, then the regular value theorem tells you that $\mathcal{O}(3)$ is a $9-6=3$ dimensional submanifold (without boundary) of $M_3(\Bbb{R})$. So, really all you have to do is calculate the derivative $Dg_A$ and show it is surjective.
If you want to think in terms of "$f$", then you look at $f: M_3(\Bbb{R}) \to \text{Sym}_3(\Bbb{R})$ defined by$f(A) = g(A) - I$, so that $\mathcal{O}(3) = f^{-1}(\{0\})$.
By the way, there is nothing special about $3$-dimensions. By considering the map $G: M_n(\Bbb{R}) \to \text{Sym}_n(\Bbb{R})$ given as $G(A):= A^tA$, and reasoning very similarly you can show that $\mathcal{O}(n)= G^{-1}(\{I\})$ is a compact, $\dfrac{n(n-1)}{2}$ dimensional submanifold of $M_n(\Bbb{R})$.