An exercise in a book I'm reading asks to describe the product of two arbitrary elements in $O(2)$. I would like to solve the exercise but I got stuck. I know that an element in $SO(2)$ can be written as
$$ B = \left ( \begin{array}{cc} \cos \theta & \sin \theta \\ - \sin \theta & \cos \theta \end{array} \right ) \in SO(2)$$
and an element in $O(2) \setminus SO(2)$ as
$$ A = \left ( \begin{array}{cc} \cos \phi & \sin \phi \\ \sin \phi & - \cos \phi \end{array} \right ) \in O(2) \setminus SO(2)$$
Is there a way of describing the product without calculating each case separately? (by that I mean calculate $AB, BA, BB'$ and $AA'$)
Based on determinant considerations, the product of two elements in $SO(2)$, or the product of two elements in $O(2)-SO(2)$, is in $SO(2)$. This should give you some direction as to what the form of the product should be. Also, computing $AB$ then computing $(AB)^{-1}$ is equivalent to computing $AB$ and $BA$, and since the inverse is easy to compute you really only have to compute $AB, BB'$, and $AA'$. Rotations are easy to multiply (just add the angles), so you just have to manually compute two products for which you know the general form of the outcome, which shouldn't be so bad.