I am given the function
$$f(x, y) = \dfrac{ax+by}{1+xy}$$
With $a, b \in \mathbb{R}$. Also it is known that the function $f(x, y)$ is a binary operation on the interval $(-1, 1)$, so $\forall$ $x, y \in (-1, 1)$ $\Rightarrow f(x, y) \in (-1, 1)$. I have to find the values of $a$ and $b$ and I am given the following options:
A. $a = b = 2$
B. $a + b \in (-1, 1)$
C. $a \in (-1, 1)$ and $b \in (-1, 1)$
D. $a = b \in [-1, 1]$
E. $a + b = 1$
Now I checked the function $f(x, y) = \dfrac{x+y}{1+xy}$ (so with $a=b=1$) and I found that this function is indeed a binary operation on the interval $(-1, 1)$. The only possible answer that contains $1$ is answer D so I know that this is the right answer. But I am curious as to how can I find that this is the answer by calculations, rather than by guessing.
Suppose $a\ne b$, and put $x=1-\epsilon,y=-1+\epsilon$ (the idea here is to make $1-xy$ very small, so that the fraction will be large). We get $$\frac{ax+by}{1+xy}=\frac{(a-b)(1-\epsilon)}{\epsilon(2-\epsilon)}$$
And if $a\ne b$, we can make this as large as we like in absolute value by choosing $\epsilon$ small enough. In particular, we can make it $> 1$ in absolute value.
Therefore $a=b$, and we can write the fraction as $$\frac{a(x+y)}{1+xy}$$
Now we need to show that $|a|\le 1$.
If $|a|>1$, then putting $x=y=1-\epsilon$ gives $$\frac{a(2-2\epsilon)}{2-2\epsilon+\epsilon^2}$$ which can be made $>1$ in absolute value by choosing $\epsilon$ small enough.
And if $|a|\le 1$, then your own work shows that the fraction is $< 1$.