I have a polynomial equation in $x$ and $y$,
$$ (a-b)(xy+1) + (ab+1)(y-x) = 0. $$ What transformation can act on $x$ and $y$ so that any point that satisfies this equation is mapped to another point that satisfies it? If there is a general way to find these transformations given a polynomial equation, that would also be nice to know.
You have $a,b,x,y\in\mathbb{R}$ ($\mathbb{R}$ can be replaced by $\mathbb{C}$ in every occurrence in this answer) such that $$\frac{a-b}{ab+1}=\frac{x-y}{xy+1}.\tag{*}$$ Let $c:=\arctan(a)$, $d:=\arctan(b)$, $u:=\arctan(x)$, and $v:=\arctan(y)$. Consider $c$, $d$, $u$, and $v$ as elements of $\mathbb{R}/\pi\mathbb{Z}$. Then, we have $$\tan(c-d)=\tan(u-v)\,.$$ Consequently, $c-d=u-v$.
Now, any transformation $T:\mathbb{R}/\pi\mathbb{Z}\to\mathbb{R}/\pi\mathbb{Z}$ of the form $$T(t)=t+q\text{ for all }t\in\mathbb{R}/\pi\mathbb{Z}\,,$$ for some fixed constant $q\in\mathbb{R}/\pi\mathbb{Z}$ satisfies $$T(t_1)-T(t_2)=t_1-t_2,$$ for all $t_1,t_2\in\mathbb{R}/\pi\mathbb{Z}$. Consider the function $f:\mathbb{R}\to\mathbb{R}$ sending $$s\mapsto \tan\big(\arctan(s)+q\big)=\frac{s+p}{1-sp}\text{ for all }s\in\mathbb{R}\,,$$ where $q\in\mathbb{R}$ is a fixed constant and $p:=\tan(q)$. Then, if $a,b,x,y\in\mathbb{R}$ satisfy (*), then $$\frac{a-b}{ab+1}=\frac{f(x)-f(y)}{f(x)\,f(y)+1}$$ too.
In fact, only functions $f$ of the given form satisfy the requirement. Thus, all solutions $(x,y)\in\mathbb{R}\times\mathbb{R}$ to (*) is of the form $$(x,y)=\left(\frac{a+p}{1-ap},\frac{b+p}{1-bp}\right)$$ for some $p\in\mathbb{R}$ such that $p\neq\dfrac{1}{a}$ and $p\neq \dfrac{1}{b}$.