The orthogonal group is defined as (with group structure inherited from $n\times n$ matrices)
$$O(n) := \{X\in \mathbb{R}^{n\times n} : X^\text{t}X=I_n\}.$$
(i) Show that $O(n)$ is an $\frac{n(n-1)}{2}$-dimensional $C^\infty$-manifold in the space of $n\times n$ matrices.
Hints. Exhibit $O(n)$ as the preimage of $0$ of the function $\phi$ from $n\times n$ matrices to the symmetric $n\times n$ matrices given by $X\mapsto X^\text{t}X − I_n$. (Note that the target space of $\phi$ is very important in order to satisfy the maximal rank condition.)
Then show that the equation $\phi'(A)H=S$ has a solution $H$ for each $A\in O(n)$ and each symmetric $n\times n$ matrix $S$. You will also need to compute the dimension of the space of symmetric $n\times n$ matrices.
(ii) Show that the tangent space $T_{I_n}O(n)$ at the identity is the space of antisymmetric matrices.
They've already given you the smooth map $\phi(X) = X^{t}X - I_n,$ and $O(n) = \phi^{-1} (0),$ so it is automatically a manifold (if the rank is constant). To see the dimension, you need to calculate the nullity of $\phi^{\prime}.$ See below for a full solution.
To calculate the tangent space at $I$, you just want to look at $\phi^{\prime} (I)$ and calculate its null space.