Prove relation $\rho:(\forall x,y \in G)(x\rho y \Leftrightarrow x * y^{-1}\in H),H\le G$ is equivalence relation
$H\le G\Rightarrow(\forall a,b\in H)a*b^{-1}\in H$
$(\forall x \in G)x\rho x$
$$x\rho x \Leftrightarrow x*x^{-1}\in H$$ $x*x^{-1}$ is an identity element under operation $*$ in $G$.
Since $H\le G$, $x*x^{-1}$ is an identity element in $H$ $\Rightarrow$ $\rho$ is reflexive.
$(\forall x,y\in G)x\rho y \Rightarrow y\rho x$ $$x\rho y \Leftrightarrow x*y^{-1}\in H$$ $$y \rho x\Leftrightarrow y*x^{-1}\in H$$ Since $x,y,x^{-1},y^{-1}\in G$, how to show that $x*y^{-1}$ and $y*x^{-1}\in H$?
$(\forall x,y,z\in G)x\rho y\land y\rho z \Rightarrow x\rho z$ $$(x*y^{-1}\land y*z^{-1})\in H\Rightarrow x*z^{-1}\in H$$
Since $y*y^{-1}$ is an identity element $\Rightarrow \rho$ is transitive.
How to prove symmetry and transitivity?
Start with $$ x \rho y \implies x * y^{-1} \in H. $$ Because $H\leq G$, this means that $(x*y^{-1})^{-1} \in H$, and the inverse is $y*x^{-1}\in H$, which means that $y\rho x$. Thus you proved symmetry. Now for the transitivity:
$$ x \rho y \ \ \& \ \ y\rho z \implies x*y^{-1},\ y*z^{-1} \in H. $$
But the operation in a subgroup is closed so
$$ (x*y^{-1})*(y*z^{-1})=x*z^{-1} \in H, $$
which means $x\rho z$, thus we have transitivity.
Also, you ask how to show that $x*y^{-1},\ y*x^{-1}$ are elements of $H$ if $x,x^{-1},y,y^{-1} \in H$. I think you maybe looking at this the wrong way. You $\textit{assume}$ that $x\rho y$, which means $x*y^{-1}\in H$, you don't have to show that. You have to show $y*x^{-1} \in H$, which is a consequence of the fact that the inverse of each element in $H$ is also in $H$, because $H$ is a subgroup of $G$.