I was trying to solve the following problem:
Check whether the transformations of the plane given below forms a Lie group. $$x^{*}=x-\varepsilon y;\quad y^{*}=y+\varepsilon x$$
I tried to identify the law of composition by taking $$x^{*}=x-\varepsilon y;\quad y^{*}=y+\varepsilon x$$ and $$x^{**}=x^{*}-\delta y^{*};\quad y^{**}=y^{*}+\delta x^{*}$$ which gives $$x^{**}=(1-\delta\varepsilon)x-(\varepsilon+\delta)y;\quad y^{**}=(1-\delta\varepsilon)y+(\varepsilon+\delta)x $$ from which I could not find the law of composition for $\varepsilon$ and $\delta$ as I don't get transformation in the same form as the original transformation.
Can I conclude that the given family of transformations does not form a Lie group? Please provide some hints and ideas. Thanks.
Try considering the group operation as $$\epsilon * \delta = \frac{\epsilon + \delta}{1-\delta \epsilon}.$$
I think it will form a group operation when $(x,y)$ is equivalent to $(x',y')$ $\iff$ $(x,y) = c \times (x',y')$.
Now $(x',y') = \left((1-\epsilon\delta)x -(\delta+\epsilon)y),(1-\epsilon\delta)x +(\delta+\epsilon)y \right)$ is equivalent to $(x'',y'') = \left(x -\frac{(\delta+\epsilon)}{(1-\epsilon\delta)}y),x +\frac{(\delta+\epsilon)}{(1-\epsilon\delta)}y\right)$.
Hence group operation is closed.