So, I know that the image for a generic point is $$\left(\frac{1-m^2}{1+m^2}x + \frac{2m}{1+m^2}(y-b), \frac{2m}{1+m^2}x - \frac{1-m^2}{1+m^2}(y-b)+b\right)$$ when you reflect it over the line $y=mx+b$. It's straightforward enough to derive that with complex mappings.
I need to derive the same result using the dot product and the projection of vectors in $\mathbb{R}^2$. I'd really appreciate any help on this, thanks!
On
First you want to translate down so that the line goes through the origin. Then the dot product of the unit vector in the direction of the line and the vector describing the point is equal to the magnitude of the projection of the vector describing the point onto the line. If you multiply that magnitude by the unit vector describing the line, you get the projection of the vector describing the point onto the line. If you subtract this from the vector describing the point, you will get the component of the vector describing the point which is perpendicular to the line. If you subtract this twice from the vector describing the point, you will get a vector describing a point which is reflected over the line. Then translate back up.
Your line can be written as $$ l(t)=(0,b)+t(1,m) $$ In other words, it is the linear space of $(1,m)$, offset by $(0,b)$. To reflect $(x,y)$ across $l(t)$, it is enough to find the orthogonal projection of $(x,y)$ onto a point $l(t)$ on the line and then to extend $$ (x,y)+2(l(t)-(x,y)). $$ The point $l(t)$ is chosen so that $l(t)-(x,y)$ is orthogonal to the direction vector $(1,m)$: $$ \left\{(0,b)+t(1,m)-(x,y)\right\}\cdot(1,m)=0. $$ So, $$ mb+t(1+m^{2})-x-my=0,\\ t = \frac{x+m(y-b)}{1+m^{2}}. $$ The projection is $$ (x,y)+2(l(t)-(x,y)) =\\ = (x,y)+2\left\{(0,b)+\frac{x+m(y-b)}{1+m^{2}}(1,m)-(x,y)\right\}\\ = \left(2\frac{x+m(y-b)}{1+m^{2}}-x,2b+2\frac{x+m(y-b)}{1+m^{2}}m-y\right)\\ = \left(\frac{1-m^{2}}{1+m^{2}}x+\frac{2m}{1+m^{2}}(y-b), \frac{2m}{1+m^{2}}x-\frac{1-m^{2}}{1+m^{2}}(y-b)+b\right) $$