Let $V = \mathbb{R}^2$ be two-dimensional Euclidean space, wit its usual $x$-coordinate and $y$-coordinate axes. Consider the linear transformation $L_\alpha: V \to V$ that performs a reflection about the line $y = \alpha x$.
(1) Write the matrix for $L_\alpha$ with respect to the basis $\mathfrak{B} = \{\textbf{e}_1, \textbf{e}_2\}$.
(2) Calculate the matrix for $L_\beta \circ L_\alpha$ (with respect to $\mathfrak{B}$) in two ways: by multiplying the matrices for $L_\beta$ and $L_\alpha$, and by determining the matrix for the resulting composed linear transformation directly.
(3) Show that the composed linear transformation $L_\beta \circ L_\alpha$ is a rotation. By what angle are vectors in $\mathbb{R}^2$ rotated under this transformation?
We know that the columns of a matrix of a linear transformation with respect to a given basis are simply the image of those basis vectors under the linear transformation. So we compute $L_\alpha(e_1)$ and $L_\alpha(e_2)$.
Let $\theta = \tan^{-1}(\alpha)$ be the angle between the positive $x$-axis and the line $y = \alpha x$. Then it is clear from geometry that$$L_\alpha(e_1) = (\cos(2\theta),\sin(2\theta))^\text{T}$$and that$$L_\alpha(e_2) = \left(\cos\left({\pi\over2} - (\pi - 2\theta)\right), \sin\left({\pi\over2} - (\pi - 2\theta)\right)\right)^\text{T} = \left(\cos\left(2\theta - {\pi\over2}\right),\sin\left(2\theta - {\pi\over2}\right)\right)^\text{T} = (\sin(2\theta), -\cos(2\theta))^\text{T}.$$So the matrix of this transformation is$$\begin{pmatrix} \cos(2\theta) & \sin(2\theta) \\ \sin(2\theta) & -\cos(2\theta)\end{pmatrix}.$$But our transformation was given in terms of $\alpha$, not $\theta$ so we use some trigonometric identities to write this matrix in terms of $\alpha$. Note that$$\cos(\theta) = {1\over{\sqrt{\alpha^2 + 1}}},\text{ }\sin(\theta) = {\alpha\over{\sqrt{\alpha^2 + 1}}}.$$Squaring these equations, we write our matrix for $L_\alpha$ as follows:$$\begin{pmatrix} \cos(2\theta) & \sin(2\theta) \\ \sin(2\theta) & -\cos(2\theta)\end{pmatrix} = {1\over{1 + \alpha^2}} \begin{pmatrix} 1 - \alpha^2 & 2\alpha \\ 2\alpha & -\alpha^2 - 1 \end{pmatrix}.$$
We begin by mutliplying matrices. This is much simpler with angle notation, so we will adopt it from now on. Let $\phi = \tan^{-1}(\beta)$. We compute$$L_\beta \circ L_\alpha = \begin{pmatrix} \cos(2\phi) & \sin(2\phi) \\ \sin(2\phi) & -\cos(2\phi)\end{pmatrix} \begin{pmatrix} \cos(2\theta) & \sin(2\theta) \\ \sin(2\theta) & -\cos(2\theta)\end{pmatrix} $$$$ = \begin{pmatrix} \cos(2\phi) \cos(2\theta) + \sin(2\phi)\sin(2\theta) & \cos(2\phi)\sin(2\theta) - \sin(2\phi)\cos(2\theta) \\ \sin(2\phi) \cos(2\theta) - \cos(2\phi)\sin(2\theta) & \cos(2\phi)\cos(2\theta) + \sin(2\phi)\sin(2\theta)\end{pmatrix}$$$$=\begin{pmatrix} \cos(2\phi - 2\theta) & -\sin(2\phi - 2\theta) \\ \sin(2\phi - 2\theta) & \cos(2\phi - 2\theta)\end{pmatrix}.$$We could, of course, use the trigonometric equalities from (1) to write this in terms of $\alpha$ and $\beta$ but there is no pressing motive to do so.
The second part of this problem asks us to compute this result directly. This does not mean with respect to an arbitrary vector $(x, y)^\text{T}$ in part because computing the image of an arbitrary vector under a composed transformation will not give us that transformation. Any algebraic computation will essentially be the matrix multiplication performed above. Thus, to do something new, and to satisfy the request in this part, we must compute the image of $e_1$ and $e_2$ geometrically as we did in (1).
Let $\theta$ and $\phi$ be as defined above. it is important to note that both of these angles are measured with respect to the positive $x$-axis so we can keep track of all of the different "pictures" that depend on the relative manitude of $\alpha$ and $\beta$. As before,$$L_\alpha(e_1) = (\cos(2\theta), \sin(2\theta))^\text{T}.$$The angle measured from this vector to the line $y = \beta x$ is $\phi - 2\theta$ (where the sign of this angle indicates the orientation as is conventional). So then the image of this vector after reflection about the line has angle $2\phi -2\theta$ with respect to the positive $x$-axis. Thus$$L_\beta \circ L_\alpha(e_1) = (\cos(2\phi -2 \theta),\sin(2\phi - 2\theta))^\text{T}.$$We perform a similar computation for $e_2$. From (1),$$L_\alpha(e_2) = \left(\cos\left(2\theta - {\pi\over2}\right),\sin\left(2\theta - {\pi\over2}\right)\right)^\text{T}$$and so has angle $2\theta - \pi/2$ with respect to the positive $x$-axis. So again$$\phi - \left(2\theta - {\pi\over2}\right) = \phi - 2\theta + {\pi\over2}$$is the angle from this vector to $y = \beta x$ and $2\phi - 2\theta + \pi/2$ is the angle of the image after reflection about this line with respect to the positive $x$-axis. Thus$$L_\beta \circ L_\alpha(e_2) = \left(\cos\left(2\phi - 2\theta + {\pi\over2}\right), \sin\left(2\phi - 2\theta + {\pi\over2}\right)\right)^\text{T} = \left(-\sin\left(2\phi - 2\theta\right), \cos\left(2\phi - 2\theta\right)\right)^\text{T}.$$So the matrix for $L_\beta \circ L_\alpha$ is$$\begin{pmatrix} \cos(2\phi - 2\theta) & -\sin(2\phi - 2\theta) \\ \sin(2\phi - 2\theta) & \cos(2\phi - 2\theta)\end{pmatrix}$$as before, verifying our previous result.
Note it is not enough to simply say that $L_\beta \circ L_\alpha$ is a rotation because it preserves lengths of vectors. For example, reflections also have this property. But we see that the matrix for $L_\beta \circ L_\alpha$ written in terms of $\theta$ and $\phi$ is in the form of a rotation matrix, and so we conclude that $L_\beta \circ L_\alpha$ is a rotation.
From our general form for rotation matrices it is clear that the angle of rotation is $2(\phi - \theta)$ or twice the angle (again with orientation) measured from $y = \alpha x$ to $y = \beta x$.