If $G$ is an odd cyclic group of order $n$, then each element $g$ of $G$ has another element $h$ such that $h^2=g$. This is because $2 x = y \mod n$ is solvable for $x$. (Note this is not the same as solving $x^2=y \mod n$.)
What other finite groups have this property?
If $G$ is a finite group of even order, then it has an element of order $2$, $a$ say. Then $a^2=e=e^2$, and the squaring map is not injective. By finiteness, the squaring map is not surjective: there are elements in $G$ which aren't squares.
If $G$ has odd order, then for each $b\in G$, $b^{(|G|+1)/2}$ is a square root of $b$.