Let $g$ be a line with equation $g:ax+by+c=0$ in Hesse normal form. I want to show that the reflection across $g$ is described by \begin{equation*}\binom{x}{y}\mapsto \binom{x}{y}-2(ax+by+c)\binom{a}{b}\end{equation*}
At the reflection across $g$ it holds the following for the image $P'$ of each point $P$:
- $P'$ lies on the perpendicular to $g$ through $P$.
- $g$ bisects $PP'$.
So to show the desired result, do we have to find the perpendicular line to $g$ ?
This reflection is an affine map, so we can represent it as linear map if we embed plane in $\mathbb R^3$ with $(x,y)\mapsto (x,y,1)$.
I will use the same idea as Robert Z, $P'$ is reflection of $P$ along the line $ax + by + c=0$ if two conditions are met:
We can describe this with linear equation $$\left\langle\frac{P+P'}2,(a,b,c) \right\rangle = 0.$$
Direction vector of the line is given by $(-b,a,0)$, so $P'-P$ must be orthogonal to it, i.e. $$\left\langle P'-P,(-b,a,0)\right\rangle = 0.$$
If we write $P = (x,y,1),\ P'=(x',y',1)$, conditions (1) and (2) can be written as
\begin{align} ax'+by'&= -ax - by - 2c\\ -bx'+ay' &= -bx+ay \end{align}
or in matrix form $$ \begin{pmatrix} a & b & 0\\ -b & a & 0\\ 0 & 0 & 1\\ \end{pmatrix}\begin{pmatrix} x'\\ y'\\ 1\\ \end{pmatrix} = \begin{pmatrix} -a & -b & -2c\\ -b & a & 0\\ 0 & 0 & 1\\ \end{pmatrix}\begin{pmatrix} x\\ y\\ 1\\ \end{pmatrix}.$$
Invert the matrix on the left hand side to get
\begin{align} \begin{pmatrix} x'\\ y'\\ 1\\ \end{pmatrix} &= \frac 1{a^2+b^2}\begin{pmatrix} a & -b & 0\\ b & a & 0\\ 0 & 0 & a^2+b^2\\ \end{pmatrix}\begin{pmatrix} -a & -b & -2c\\ -b & a & 0\\ 0 & 0 & 1\\ \end{pmatrix}\begin{pmatrix} x\\ y\\ 1\\ \end{pmatrix}\\ &= \frac {-1}{a^2+b^2}\begin{pmatrix} a^2-b^2 & 2ab & 2ac\\ 2ab & b^2-a^2 & 2bc\\ 0 & 0 & -(a^2+b^2)\\ \end{pmatrix}\begin{pmatrix} x\\ y\\ 1\\ \end{pmatrix} \end{align}
or if you want to get back to $\mathbb R^2$, we get affine map
$$\begin{pmatrix} x\\ y\end{pmatrix} \mapsto \frac {-1}{a^2+b^2}\begin{pmatrix} a^2-b^2 & 2ab \\ 2ab & b^2-a^2 \end{pmatrix}\begin{pmatrix} x\\ y \end{pmatrix}-\frac {1}{a^2+b^2}\begin{pmatrix} 2ac\\ 2bc\end{pmatrix}.$$