Let $f:GL(n,\mathbb{R})\to GL(n,\mathbb{R})$ be the smooth map $A\mapsto A^TA$. Observe that $f$ has constant rank on $GL(n,\mathbb{R})$ by chain rule and that $O(n,\mathbb{R})$ is the preimage of $I$, therefore is a regular submanifold.
To compute the dimension of $O(n,\mathbb{R})$, it thus suffices to compute the rank of $f$ at any point, e.g at $I$. In an appropriate choice of bases, the matrix of $Df|_{I}$ is given by $$\left[\frac{\partial f^{ij}}{\partial x^{kl}}|_I\right]$$ where $(i,j)$ with $1\le i,j\le n$ gives the row index and $(k,l)$ $1\le k,l\le n$ gives the column index.
So how do I proceed from here really?
The dimension of $O(n)$ and $SO(n)$ is $\frac{n^2-n}{2}$. This is best seen by computing their Lie algebra $\mathfrak{so}(n)$, which obviously has vector space dimension $\frac{n^2-n}{2}$. For details. see for example here. See also the answer in the question Show that an orthogonal group is a $\frac{n(n−1)]}2-$dim. $C^\infty$-Manifold and find its tangent space.