Over $\mathbb{R}$, $aRb \iff a^2 - b^2 = 1$.
I tried determining if it was antisymmetric. I seem to have done it, but while doing the equation, I stumbled upon a scenario that always made me doubt my decisions:
Have
$$aRb \land bRa$$
$$a^2 - b^2 = 1 \land b^2 - a^2 = 1$$
$$-a^2 + b^2 = -1 \land b^2 - a^2 = 1$$
$$-2a^2 + 2b^2 = 0$$
$$\color{#C00}{2(-a^2 + b^2) = 0}$$
$$\color{#C00}{-a^2 + b^2 = 0}$$
$$b^2 = a^2$$
$$b = a$$
I passed the $2$ to divide to the other side. The other side has a $0$, so the $2$ is essentially killed off with no apparent consequence.
Is this equation correct?
The deduction you have does not work, as $a^2 = b^2$ does not imply $a = b$. It is actually much simpler. Assuming $aRb$ and $bRa$, you get (third line of equations) $-a^2 + b^2 = -1$ and $b^2 - a^2 = 1$. So $b^2 - a^2$ must be equal to $1$ and $-1$ at the same time, which is not possible.