I found that $\big(\mathcal{O}(n), \times \big)$ and $\big(\mathcal{GL}_n(\mathbb{R}), \times \big)$ are not isomorphic, because their centers are not (respectively $\big \{\pm I_n \big \}$ and $\mathbb{R}^*I_n$).
Then I wondered whether $\mathcal{SO}(n)$ and $\mathcal{O}(n)$ are isomorphic.
By considering the centers once again, I found the solution for $n$ odd, but I don't know if much is known about the center of $\mathcal{SO}(n)$ for $n$ even and $n \ge 4$...
Is there any simple argument to determine whether $\mathcal{O}(n)$ and $\mathcal{SO}(n)$ are isomorphic ?
Indeed, for $n$ odd, the center of $SO(n)$ is trivial, while the center of $O(n)$ is non-trivial. If $n\ge 4$ is even, however, the center of $SO(n)$ and $O(n)$ are both given by $\mathbb{Z}/2$. So we cannot distinguish $O(n)$ and $SO(n)$ as abstract groups this way. We may look at the derived groups instead, as indicated in the comments (but you said that you cannot use this concept).
It is certainly much more natural to consider these groups as Lie groups or algebraic groups. Then it is easy to distinguish them (use connectedness). As abstract groups, the orthogonal groups are very complicated, e.g., its automorphism group, or its cohomology. For example, $SO(2)$ has uncountably many outer automorphisms as an abstract group.