Let $G$ be an abelian group with Identity $e$. Let $H$ be a subset of $G$ such that each of its elements square is the identity element.
We now prove that $H$ is a subgroup of $G$: Clearly $e \in H$ since $e^2=e$. Also if $a \in H$ then $(a^{-1})^2=e$ which implies that inverse of each element of $H$ will be in $H$. There is one more thing to show - closure, to do that we observe that for any $a$ and $b$ in $H$, $(ab)^2= (ab)(ab)= a(ba)b = a(ab)b =(aa)(bb)=e$ . So the operation is closed in $H$.
It is a good observation to make note how the condition- $G$ is Abelian helped us to prove that $H$ is a subgroup.
Now what if $G$ is not Abelian?
A quick search yields an example of non Abelian group which doesn't satisfy this condition. If we consider $G$ as set of all $2 \times2$ matrices with nonzero determinant is sufficient to show this. Although I didn't prove it, I believe that if $G$ is any group of $n \times n$ matrices with nonzero determinant, then $H$ will not be a subgroup of $G$.
Is there any group which is non isomorphic to these group of matrices such that $H$ is not a subgroup?
Hint: Try the smallest nonabelian group: the dihedral group of order $6$.
NB: This group is isomorphic to the subgroup of ${\rm GL}_2(\Bbb Z)$ generated by $$\begin{pmatrix} 0 & 1\\ -1 & -1\end{pmatrix}\text{ and }\begin{pmatrix} 1 & 0\\ -1 & -1\end{pmatrix},$$ so it might not be in the spirit of your question.