If one of the straight lines given by equation $$ax^2 + 2hxy + by^2 = 0$$ coincide with one of those given by $$a_2x^2 + 2h_2xy + b_2y^2 = 0$$ and the other represented by them be perpendicular, prove that $${ha_2b_2\over b_2 - a_2} = {h_2ab\over b - a} = \sqrt{-aa_2bb_2} \tag{+}$$ All straight lines pass through origin.
Let the inclination of four lines be $m, m_2, m_3, m_4$
Now $m = m_3$ and $\displaystyle m_2 = {-1\over m_4}$
$ax^2 + 2hxy + by^2 = 0$ can be represented as $$b(y - mx)(y - m_2x) = 0$$
On expanding we get
$$b(y^2 - xy(m + m_2) + mm_2x^2) = 0$$
From which we get $-b(m + m_2) = 2h$ and $bmm_2 = a$.
Also $a_2x^2 + 2h_2xy + b_2y^2 = 0$ can be represented as $b_2(y- m_3x)(y - m_4x) = 0$
Which is same as $$b_2(y- mx)\left(y + {1\over m_2}x\right) = 0$$
On doing same procedure as we did with first pair of lines,
we get $\displaystyle 2h_2 = \left({1\over m_2} - m\right)b_2$ and $\displaystyle {-bm\over m_2} = a_2$
On substituting these values in (+) we get $${ha_2b_2\over b_2 - a_2} = {h_2ab\over b - a} = {bb_2m\over 2}$$
but $\displaystyle \sqrt{-aa_2bb_2} = b_2bm$
From this I am getting $$\sqrt{-aa_2bb_2} \ne {ha_2b_2\over b_2 - a_2} = {h_2ab\over b - a}$$.
Where did I go wrong ?
You did nothing wrong, and $(+)$ does not hold in general.
We have
$$-b(m + m_2) = 2h\quad\text{and}\quad bmm_2 = a\quad\text{and}\quad 2h_2 = \left({1\over m_2} - m\right)b_2\quad\text{and}\quad {-b_2m\over m_2} = a_2$$
So, we get $$\frac{ha_2b_2}{b_2-a_2}=\frac{\dfrac{-b(m+m_2)}{2}\cdot \dfrac{-b_2m}{m_2}\cdot b_2}{b_2-\left(-\dfrac{b_2m}{m_2}\right)}=\frac{bb_2m}{2}$$
and $$\frac{h_2ba}{b-a}=\frac{\dfrac{(\frac{1}{m_2}-m)b_2}{2}\cdot b\cdot bmm_2}{b-bmm_2}=\frac{bb_2m}{2}$$ from which $$\frac{ha_2b_2}{b_2-a_2}=\frac{h_2ab}{b-a}$$ follows.
However, $$\frac{h_2ab}{b-a}=\sqrt{-aa_2bb_2}$$ does not hold since $$\left(\frac{h_2ab}{b-a}\right)^2=\left(\frac{b_2bm}{2}\right)^2=\frac{-aa_2bb_2}{\color{red}{4}}$$