What is wrong in this proof?
In $\triangle ABC$ with right angle at $B$, $BD$ is drawn as an altitude on $AC$. Let $AB=a$ and $BC=b$. So by similarity, $AD=\frac{a^2}{\sqrt{a^2+b^2}}$ and similarly $CD=\frac{b^2}{\sqrt{a^2+b^2}}$. Now, $\triangle ABD,BDC$ are also right angled at $D$. So applying the Pythagorean theorem, $$BD^2=a^2-\frac{a^2}{\sqrt{a^2+b^2}}=b^2-\frac{a^2}{\sqrt{a^2+b^2}}$$ Solve these two equations as: $$a^2-b^2=a^2-\frac{b^2}{\sqrt{a^2+b^2}}$$ So $\sqrt{a^2+b^2}=1$. That means for all values of $a$ and $b$, $a^2+b^2=1$.
But this is absurd.
Your problem is actually not even really a misuse of a rule. It's a typo.
This should be:
$$BD^2=a^2-\frac{a^2}{\sqrt{a^2+b^2}}=b^2-\frac{b^2}{\sqrt{a^2+b^2}}$$
Reducing that down, we get:
$$(a^2)\sqrt{a^2+b^2}-a^2=(b^2)\sqrt{a^2+b^2}-b^2$$
Factoring...
$$(a^2)\left(\sqrt{a^2+b^2}-1\right)=(b^2)\left(\sqrt{a^2+b^2}-1\right)\\ a^2=b^2$$