I need to understand the definition of subcoset. Let $G$ be a group and $x \in G$. I google and get the definition that " A subcoset of $Gx$ is a subset of $Gx$ that is either empty or a coset of a subgroup of $G$"
I still not able to understand the definition fully. What is its significance ? Is there an good example to understand it.
Probably you mean that $G$ is a group, $H$ is a subgroup of $G$ and $x\in G$.
Then a subset $T$ of $Hx$ is a subcoset of $Hx$ if either $T=\emptyset$ or there exist a subgroup $K$ of $G$ and $y\in G$ with $T=Ky$.
For instance, $Kx$ is a subcoset of $Hx$ for every subgroup $K\subseteq H$, but there can be subcosets that are not of this form. Any singleton in $Hx$ is a subcoset: if $y\in Hx$, then $\{y\}=\{1\}y$.
As far as I remember, there's ample literature on the study of groups through the set of their subcosets.