For each angle $\theta \in \mathbb{R}$, we get two corresponding matrices:
$$\mathrm{Rot}^\theta = \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos\theta\end{bmatrix}, \qquad \mathrm{Refl}^\theta = \begin{bmatrix} \cos 2\theta & \sin 2\theta \\ \sin 2\theta & -\cos2\theta\end{bmatrix}.$$
The associated linear transformations:
Rotate by an angle $\theta$ about the origin
Reflect in the line through the origin at an angle $\theta$ from the $x$-axis
Now another way of thinking about $\mathrm{Refl}^\theta$ is that it decomposes each vector into two components, a parallel and perpendicular component, then scales the perpendicular component by $-1$, and adds the resulting vectors. This suggests that there should be a generalization of $\mathrm{Refl}^\theta$ that we'll denote $\mathrm{Dil}^{\theta}_a$ in which we dilate by a factor of $a$ perpendicular to the relevant line. It ought to be the the case that $\mathrm{Refl}^{\theta} = \mathrm{Dil}^\theta_{-1}.$
Question. What matrix decribes $\mathrm{Dil}^\theta_a$?
In case $x$-axis is the reflecting line the matrix you are seeking is $A=\pmatrix {1 & 0\cr 0 & a\cr}$. For a reflecting line that makes angle $\theta$ with $x$-axis the matrix would be got by change-of-basis formula: $ \mathrm{Rot}^\theta * A *\mathrm{Rot}^{-\theta}$. (here $*$ denotes matrix multiplication}