I'm not sure why $O(n)\subseteq R^{n\times n}$, the set of real orthogonal matrices of order $n$, is compact. I can see why it's bounded, but I still fail to understand why this would be a closed set.
In the notes I have, the author says this is since $$x_{i1}y_{j1} + \ldots + x_{in}y_{jn} = \delta_{ij}$$ for $1\leq i,j\leq n$ and thus $O(n)$ can be seen as a closed subset of $R^{n^2}$. I just don't see why this shows it's closed though.
As an easier example, do you see why the equation $$ C: x^2 + y^2 = 1$$ determines a closed subset of $\mathbb R^2$?
One way to explain this, as mentioned in the comments, is to notice that the map $$\mathbb R^2 \to \mathbb R$$ $$ (x,y) \mapsto x^2 + y^2$$ is continuous, so the pre-image of the closed set $\{1\}\subset\mathbb R$ is a closed set in $\mathbb R^2$.
But this explanation hides the important step which is noticing why this map is continuous. So instead we could argue directly why the circle described above is closed. This is equivalent to showing that the complement $$ U: x^2 + y^2 \neq 1$$ is an open subset of $\mathbb R^2$. Suppose we pick a point $(x_0, y_0)$ inside $U$, say $x_0^2 + y_0^2 = 2$. Then I claim that for sufficiently small $\epsilon_1, \epsilon_2$, say $|\epsilon_i|<M$ for some positive constant $M$, we can guarantee that $$ (x_0+\epsilon_1)^2 + (y_0 + \epsilon_2)^2 > 1$$ which implies that an open neighborhood of $(x_0,y_0)$ also lies in $U$. This shows that $U$ is an open in $\mathbb R^2$. I'll leave it up to you to figure out which constant $M$ would satisfy the claim made above.