The inverse of $R$ is the relation $R^{-1}$ from $B$ to $A$ and is defined as $R^{-1}=\{(b,a)\in B\times A|(a,b) \in R\}$.
Let $G=\{(x,y)\in R \times R | x>y\}$.
Then, in How to Prove It,
$G^{-1}=\{(x,y)\in R \times R | (y,x)\in G\}$ (I'm confused here since it's of a different form from the definition given)
$G^{-1}=\{(x,y)\in R \times R | y>x \}$
$G^{-1}=\{(x,y)\in R \times R | x<y \}$
But according to the definition, shouldn't it be
$G^{-1}=\{(y,x)\in R \times R | (x,y) \in G \}$
$G^{-1}=\{(y,x)\in R \times R | x >y\}$
...
or should it be $G^{-1}=\{(y,x)\in R \times R | x<y\}$ since we are taking the $x$ and respective $y$ co-ordinates from $G$ and switching their them in the ordered pair $(y,x)$?
I'm not sure where to go from here.
What you have is the same thing, fundamentally, and you've simply labeled them differently.
To help distinguish, let's let $$\newcommand{\R}{\mathbb{R}} G^{-1}_{1} =\{(x,y)\in \R \times \R \mid x<y \} $$ and $$ G^{-1}_{2} =\{(y',x')\in \R \times \R \mid y'<x'\} $$
Goal: prove these are equal sets, i.e. $G^{-1}_{1} = G^{-1}_{2}$.
Take a pair $(x,y) \in G_{1}^{-1}$, and let $y' = x$ and $x' = y$, and hopefully the proof that $(x,y) \in G_2^{-1}$ (and the proof of the reverse inclusion) is immediate.
It should be $$ G^{-1} = \{ (y,x) \in \R \times \R \mid x > y \} $$ since the condition in the previous line is that $(x,y) \in G$, and we know from the definition of $G$ that $(x,y) \in G \iff x>y$.
The switching of coordinates only happens in the "first half" of the setbuilder notation as you notice in the opening: $$ (a,b) \in R \iff (b,a) \in R^{-1} $$