Determine a matrix $M\in\mathbb{R}^{3\times 3}$ that describes a transformation (based on homogenous coordinates) that reflects through the line that passes through the points $P(2|-4)$ and $Q(-6|2)$.
The idea is to combine the following:
- A translation by $(-2|4)$ s.t. $P$ is translated into the origin.
- A rotation s.t. the translated point $Q'$ is rotated onto the $x$-axis.
- A reflection through the $x$-axis.
- Reversing the rotation and translation.
Eventually this will yield $M=T'\cdot R' \cdot X \cdot R\cdot T$ with the following matrices:
$$T=\begin{pmatrix}1&0&-2\\ 0&1&4 \\ 0&0&1\end{pmatrix},\quad T'=\begin{pmatrix}1&0&2\\ 0&1&-4 \\ 0&0&1\end{pmatrix},\quad X=\begin{pmatrix}1&0&0\\ 0&-1&0 \\ 0&0&1\end{pmatrix}.$$
In order to figure out the angle for the rotation consider the slope from $P'$ to $Q'$ and the hypotenuse with length 10. Therefore $\sin(\alpha)=6/10$ and $\cos(\alpha)=8/10$. Those values imply
$$R=\begin{pmatrix}8&-6&0\\ 6&8&0 \\ 0&0&10\end{pmatrix},\quad R'=\begin{pmatrix}8&6&0\\ -6&8&0 \\ 0&0&10\end{pmatrix}.$$
Unfortunately the solution to this exercise differs from mine as it states that
$$R=\begin{pmatrix}-8&6&0\\ -6&-8&0 \\ 0&0&10\end{pmatrix},\quad R'=\begin{pmatrix}-8&-6&0\\ 6&-8&0 \\ 0&0&10\end{pmatrix}.$$
I'd like to know whether you can explain what's the difference and what I am doing wrong.
You’re not doing anything wrong per se. Your rotation maps $Q'$ to a point on the negative $x$-axis; the book solution instead maps it to the positive $x$-axis. Both lead to the same matrix $M$ in the end, as you can verify for yourself by multiplying out $R'TR$ for both the book’s and your chosen rotations.