I can't figure out the rule for the composition of the transformation done to get the image $∆B"C"D"$. At first I tried translating $D"$ so that it aligned to $D$ and reflected it to get the rule $r_{x=-3}\circ T_{(0,-2)}$. But this rule doesn't map $C$ and $B$ to $C"$ and $B"$. Any ideas how to figure out the rule?
First translate one of the three points to its final destination. We are given
$B = (-1, 3), C = (3, 1), D = (2, 4) $
We're also given,
$B'' = (-7,5), C'' = (-5,1), D'' = (-8,2) $
So the translation will define the image of $P$ to be $P'$ as follows
$P' = P + (B'' - B) = P + (-6, 2) $
After the translation is made, the images of $B, C, D$ are
$B' = B" = (-7,5) $
$C' = C + (-6,2) = (-3, 3)$
$D' = D + (-6,2) = (-4, 6)$
Now we'll apply an affine transformation, such that the image of $B'$ is $B"$, and the image of $C'$ is $C''$ and the image of $D'$ is $D''$. Such a transformation is given by
$P'' = B' + T (P' - B') $
Note that the image of $B'$ is automatically itself. Now, let's plug in the other two points:
$C'' = (-5, 1) = (-7, 5) + T ( (-3,3) - (-7, 5) ) $
Simplifying,
$ (2, -4) = T (4, -2) $
Similarly
$D'' = (-8, 2) = (-7, 5) + T ( (-4,6) - (-7, 5) ) $
Simplifying,
$(-1, -3) = T (3, 1) $
In matrix form, the above two equations are:
$\begin{bmatrix} 2 && -1 \\ -4 && - 3 \end{bmatrix} = T \begin{bmatrix} 4 && 3 \\ -2 && 1 \end{bmatrix}$
Hence,
$T =\begin{bmatrix} 2 && -1 \\ -4 && - 3 \end{bmatrix} \begin{bmatrix} 4 && 3 \\ -2 && 1 \end{bmatrix}^{-1} =\dfrac{1}{10} \begin{bmatrix} 2 && -1 \\ -4 && - 3 \end{bmatrix} \begin{bmatrix} 1 && -3 \\ 2 && 4 \end{bmatrix} $
Carrying out the matrix multiplication, gives us
$T = \begin{bmatrix} 0 && -1 \\ -1 && 0 \end{bmatrix} $
Now, we know that a rotation matrix in two dimensions is given by,
$R = \begin{bmatrix} \cos \theta && - \sin \theta \\ \sin \theta && \cos \theta \end{bmatrix} $
Clearly, it is not possible for $T$ to be a rotation matrix, so it has to be a combination of reflection and rotation (or just a reflection).
The reflection matrix about the origin in the line $u^T \cdot (x, y) = 0 $, where $u $ is a unit vector, i.e. $u = (\cos \phi, \sin \phi) $, is given by
$F = I - 2 u u^T = \begin{bmatrix} - \cos 2 \phi && - \sin 2 \phi \\ -\sin 2 \phi && \cos 2 \phi \end{bmatrix} $
Inspection of the matrix $T$ that we obtained above, and the expression for matrix $F$, we deduce that if we set $\phi = \dfrac{\pi}{4} $ , then $F = T$.
Hence, we conclude our transformation is a composition of a translation followed by a reflection. If our translation is selected to send $B$ to $B''$, then the reflection is about the line passing through point $B''$ and whose unit normal vector is $\dfrac{1}{\sqrt{2}} (1, 1) $, a simple calculation gives the equation of the line as $x + y = -2$.
Note that this is not a unique way of constructing the transformation $T$. We could have, combined a rotation and reflection, and written $T = RF $ , then deduced possible values for the rotation angle $\theta $ and the reflection parameter $\phi$.