Question: Let $(G, \circ)$ be a group and $H$ be a non-empty subset of $G$. A relation $\rho$ defined on $G$ by $$a\,\rho\ b\quad \text{if and only if}\quad a\circ b^{-1}\in H$$ for $a,b\in G$, is an equivalence relation on $G$. Prove that $(H,\circ)$ is a subgroup of $(G, \circ)$.
Progress: To show $(H, \circ)$ is a sub-group it is enough to show $a,b\in H\implies a\circ b^{-1}\in H$. How can I show this?
On
You probably have already seen that if $H$ is a subgroup, then $\rho$ is an equivalence relation. The proof uses the properties of subgroups:
The relation $\rho$ is reflexive: indeed, if $a\in G$, then $aa^{-1}=e\in H$ by 1, so $a\mathrel{\rho}a$
The relation $\rho$ is symmetric: indeed, if $a\mathrel{\rho}b$, we have $ab^{-1}\in H$, so $(ab^{-1})^{-1}=ba^{-1}\in H$, for 2; therefore $b\mathrel{\rho}a$.
The relation $\rho$ is transitive: indeed, if $a\mathrel{\rho}b$ and $b\mathrel{\rho}c$, then $ab^{-1}\in H$ and $bc^{-1}\in H$, so $(ab^{-1})(bc^{-1})=ac^{-1}\in H$ by 3; therefore $a\mathrel{\rho}c$.
The converse proof follows a very similar pattern.
Since $\rho$ is reflexive, we have $e\mathrel{\rho}e$, that is, $ee^{-1}=e\in H$.
If $a\in H$, then $ae^{-1}=a\in H$; therefore $a\mathrel{\rho}e$ and, since the relation is symmetric, also $e\mathrel{\rho}a$, which means $ea^{-1}=a^{-1}\in H$.
If $a,b\in H$, we have $a=(ab)b^{-1}\in H$ and therefore $ab\mathrel{\rho}b$; moreover $b\mathrel{\rho}e$, so $ab\mathrel{\rho}e$ by transitivity; hence $ab=abe^{-1}\in H$.
We first show that $e\in H$. By the reflexivity of $\rho$, $e\rho e$, and hence, $e=e\circ e^{-1}\in H$.
Next, we show that $a,b\in H\implies a\circ b^{-1}\in H$. Assume $a, b\in H$. Then, $a\circ e^{-1}= a\in H$ and $b\circ e^{-1}=b\in H$. By assumption, it follows that $a\rho e$ and $b\rho e$. Since $\rho$ is symmetric, we have $e\rho b$, and hence, by transitivity, $a\rho b$, as well. Therefore, $a\circ b^{-1}\in H$. This is what is required to show that $H$ is a subgroup of $G$.