Using vectors I tried obtain the expression for reflection of a ray line $ax+by+c=0$ reflected by a mirror whose normal is given by $a'x+b'y+c'=0$.
The point of intersection is $$X=\frac{bc'-cb'}{ab'-ba'} $$ and $$Y= \frac{ca'-ac'}{ab'-ba'}$$.
The incident vector $$\hat I=\frac{b}{a^2+b^2}\hat i-\frac{a}{a^2+b^2}\hat j $$
The normal vector $$\hat n=\frac{b'}{a'^2+b'^2}\hat i-\frac{a'}{a'^2+b'^2}\hat j
$$
The reflected vector is given by $\hat r=\hat I-2(\hat I.\hat n)\hat n$
After this the equations become too big. Is their another elegant way to get this result and remember it ? Or is it advisable to follow the method for each individual problem ?
One of the simplest methods is by reasoning on line bissectors.
The basic observation is that the incident ray and the reflected ray have the mirror as their internal or external bissector line (internal or external according to the angle of incidence of the ray on the mirror: $ \leq \pi/4$ or not).
In this case, it is important to work with the normalized form of the equation of a straight line: $ax+by+c=0$ with $a^2+b^2=1$.
The equations of the 2 line bissectors (internal and external) of two straight lines with resp. normalized equations $ax+by+c=0$ and $\alpha x+\beta y+\gamma=0$ are (see explanation below):
$$ax+by+c=s(\alpha x+\beta y+\gamma) \ \ \text{(where} \ \ s=\pm 1 \text{)} \ \ \text{due to coincide with} \ \ a'x+b'y+c'=0$$
which gives the coefficients:
$$a'=a-s\alpha, \ \ b'=b-s\beta, \ \ c'= c-s\gamma \ \ \ (1)$$
Which $s$ one must take in (1)? It entirely depends on the angle of incidence, or more precisely of its cosine, as said at the beginning, a condition that can be checked by the value taken by the dot product of the unit normals $(a,b)$ and $(a',b')/\|(a',b')\|$.
Explanation: Let $(D)$ be a straight line with normalized equation $ax+by+c=0$; then $d=|ax_1+by_1+c|$ is the distance of point $(x_1,y_1)$ to $(D)$ http://www.cut-the-knot.org/Curriculum/Calculus/DistanceToLine.shtml. Thus, if ($\Delta$) is another straight line with normalized equation $\alpha x+ \beta y+\gamma =0$, condition $|ax+by+c|=|\alpha x+ \beta y+\gamma|$ expresses that point $(x,y)$ is at the same distance of $(D)$ and $(\Delta)$, i.e., belongs to one of the two line bissectors.
Recall:
A straight line with equation $ax+by+c=0$ has $(a,b)$ as a normal vector.
the normalized version of equation $ax+by+c=0$ is obtained by dividing it by $\sqrt{a^2+b^2}$.