The question: Given a bijection $f:\mathbb{R}^2\to\mathbb{R}^2,\, (x,y)\mapsto (2x+y-m, x-2y-m)$. Determine $f(A)$ and $f^{-1}(A)$ where $A=\{(x_1,x_2)\in\mathbb{R}^2 \mid (x_1+m)^2-(x_2-m)^2=1\}$.
My attempt: So the condition of $A$ can be interpreted as: $$ \begin{cases} x=\sqrt{1+(y-m)^2}-m&\qquad&\text{if } x>-m\\ x=-m-\sqrt{1+(y-m)^2}&\qquad&\text{if } x\le -m\\ \end{cases} $$ But then substitute into the bijection would make things ugly. Beside I feel that there is some logical flaw with such argument. Any suggestion on how to deal with this?
Your function and its inverse act as
$$f(x,y) = (2x+y-m, x-2y-m)$$ $$f^{-1}(x,y) = \frac15(2x+y+3m, x-2y-m)$$
Set $(u,v) = f(x,y)$, then $(x,y) = f^{-1}(u,v) = \frac15(2u+v+3m, u-2v-m)$. Therefore:
\begin{align} (u,v) \in f(A) &\iff (x,y) \in A\\ &\iff 1 = (x+m)^2 - (y-m)^2\\ &\iff 1 = \left(\frac{2u+v+3m}{5}+m\right)^2 - \left(\frac{u-2v-m}{5}-m\right)^2\\ &\iff 1 = \frac{1}{25} (28 m^2 + 44 m u + 3 u^2 - 8 m v + 8 u v - 3 v^2)\\ &\iff 25 = (2 m + 3 u - v) (14 m + u + 3 v) \end{align}
Therefore $$f(A) = \{(u,v) \in \mathbb{R}^2 : (3 u - v+2m) (u + 3 v+14m) = 25\}$$
Set $(x,y) = f^{-1}(u,v)$, then $(u,v) = f(x,y) = (2x+y-m, x-2y-m)$. Therefore
\begin{align} (x,y) \in f^{-1}(A) &\iff (u,v) \in A\\ &\iff 1 = (u+m)^2 - (v-m)^2\\ &\iff 1 = \left(2x+y-m+m\right)^2 - \left(x-2y-m-m\right)^2\\ &\iff 1 = -4 m^2 + 4 m x + 3 x^2 - 8 m y + 8 x y - 3 y^2\\ &\iff 1 = -(2 m - 3 x + y) (2 m + x + 3 y) \end{align}
Therefore $$f^{-1}(A) = \{(x,y) \in \mathbb{R}^2 : (3 x - y-2m) (x + 3 y+ 2 m ) = 1\}$$