Let $G=\langle x, y \mid x^4=y^3=1, y^{-1}xy=x^{-1}\rangle$. What is $G$?
I started by taking
$$y^2=y^{-1}xy y^{-1}x^{-1}= (y^{-1}xy) y^{-1}x^{-1}=x^{-1}y^{-1}x^{-1}=x^{-1}y^{-1}(y^{-1}xy)=x^{-1}yxy$$ which implies that $y=x^{-1}yx$ and hence $xy=yx$, i.e $G $ is abelian! But actually the question is to show that this group is of order $12$. So if my solution is not correct, what is $G$?
There is an error in the question.
As your computation shows, $x$ and $y$ commute (i.e. $G$ is abelian). As such, the relation $y^{-1}xy = x^{-1}$ simplifies to $x = x^{-1}$, so $x$ actually satisfies $x^2 = 1$. Therefore, an alternative presentation for $G$ is
$$\langle x, y \mid x^2 = y^3 = 1, xy = yx\rangle$$
from which it follows that $G \cong \mathbb{Z}_2\times\mathbb{Z}_3$; in particular, $G$ has order $6$, not $12$.
This can be confirmed using the Magma Online Calculator by using the following code
which computes the order of the group (and returns $6$).