Question: Let $G$ be a group and $H$ be a nonempty subset of $G$. A relation $\rho$ defined on $G$ by ``$a\rho b$ if and only if $a\circ b^{-1}\in H$" for $a,b\in G$, is an equivalence relation on $G$. Prove that $(H,\circ)$ is a subgroup of $(G, \circ)$.
Answer: Let $a\in G$, then $a\rho a $ holds and so $a\circ a^{-1}=e\in H$
So $H$ contains identity element.
I don't know whether my approach is right or not. How can I show that $H$ is a subgroup of $G$.
Thaks
On
You are starting out good. Now, suppose that $a\in H$. Then you get $a\circ e^{-1}=a \in H$. By symmetry, $e\circ a^{-1}=a^{-1}\in H.$
Then all you have to do is closure under multiplication, which is transitivity.
On
You are on the right way.
You already proved that reflexivity of the equivalence relation implies that $e\in H$. Similarly other properties of equivalence relations imply that $H$ is a subgroup:
Since $a\rho b\Leftrightarrow b\rho a$, we have $a\circ b^{-1} \in H \Leftrightarrow b \circ a^{-1} = (a \circ b^{-1})^{-1} \in H$, so $H$ is closed under taking inverses (note that any element $h\in H$ can be represented as a product $a\circ b^{-1}$, just let $a=h$ and $b=e$).
Now, let $x,y \in H$. Let $a=x$, $b=e$, $c=y^{-1}$. Transitivity $a\rho b \land b\rho c \Rightarrow a\rho c$ implies $a\circ b^{-1} = x \in H \land b\circ c^{-1} = y \in H \Rightarrow a\circ c^{-1} = x\circ y \in H$, so $H$ is closed under group operation.
It's a good first step, the next step is to check if $ab\in H$ and that inverses exist. These can be done at once by checking if $ab^{-1}\in H$ which we can easily do because if $a,b\in H$ then $a\rho b$ which means that $ab^{-1}\in H$ by definition of the relation, you're done.