I'm trying to show $d\mu=\frac{dx\space dy}{y^2}$ is an invariant volume element under the action of $SL(2,\mathbb{R})$. Since I know $SL(2,\mathbb{R})$ is generated by $ \begin{bmatrix} a&\\ &a^{-1} \end{bmatrix}, \begin{bmatrix} 1&b\\ &1 \end{bmatrix}, \begin{bmatrix} &-1\\ 1& \end{bmatrix} $, I only need to show that the given volume element is invariant under these actions. I checked for the first two but the last one is giving me some trouble.
For $S= \begin{bmatrix} &-1\\ 1& \end{bmatrix}$, $Sz=-\frac{1}{z}=-\frac{\overline{z}}{|z|^2}=-\frac{x-iy}{|z|^2}.$ So we have $x\to-\frac{x}{|z|^2}$ and $y\to\frac{y}{|z|^2}.$ So $S$ takes $|y|^{-2}dx\,dy\to |z|^4|y|^{-2}d\left(\frac{-x}{|z|^2}\right)d\left(\frac{y}{|z|^2}\right)=-|y|^{-2}dx\,dy.$ Why am I off by $-1$?
I'd rather do the calculation in complex coordinates altogether. If $w=f(z)=-1/z$, then $$f^*\frac{\frac i2 dw\wedge d\bar w}{\text{Im}(w)^2} = \frac{\frac i2 \frac{dz}{z^2}\wedge \frac{d\bar z}{\bar z^2}}{\frac{\text{Im}(z)^2}{|z|^4}} = \frac{\frac i2 dz\wedge d\bar z}{\text{Im}(z)^2},$$ as desired.
Your computation is quite suspect, in that you never computed $d(\frac x{|z|^2}) = d(\frac x{x^2+y^2})$. Indeed, \begin{align*} d\big({-}\frac x{x^2+y^2}\big)\wedge d\big(\frac y{x^2+y^2}\big) &= \frac{((x^2-y^2)dx+2xy\,dy)\wedge(-2xy\,dx+(x^2-y^2)dy)}{(x^2+y^2)^4}\\&=\frac{\big((x^2-y^2)^2+4x^2y^2\big)dx\wedge dy}{|z|^8} = \frac{dx\wedge dy}{|z|^4}, \end{align*} and it works out perfectly. In particular, let me emphasize that you need to be computing with differential forms here. As a tangential comment, let me say that $SL(2,\Bbb R)$ acts by orientation-preserving transformations, and so you cannot end up changing the sign of the area $2$-form.