Consider the geometry below, where the small circle is touching both semi circles of radius $5$ and the side of the square. Find the radius of the small circle.
My try: $M$ and $N$ are centers of semicircles and $G$ is center of small circle indicated below.Let $r$ be the radius of small circle.
We have $$\beta=\frac{3\pi}{4}-\alpha$$
So we have
$$\sin \beta=\frac{\sin \alpha+\cos \alpha}{\sqrt{2}}$$
Also $$\sin \beta=\frac{r}{5-r}=\frac{\sin \alpha+\cos \alpha}{\sqrt{2}}\tag{1}$$
By cosine law $$\cos \alpha=\frac{(5-r)^2+(5 \sqrt{2})^2-(5+r)^2}{2(5 \sqrt{2})(5-r)}=\frac{5-2 r}{\sqrt{2}(5-r)}\:\Rightarrow \sin \alpha=\sqrt{\frac{2(5-r)^2-(5-2 r)^2}{2(5-r)^2}}=\frac{\sqrt{25-2r^2}}{\sqrt{2}(5-r)}$$
Hence from $(1)$ we get $$\frac{(5-2 r)+\sqrt{-2 r^2+25}}{\sqrt{2}(5-r)}=\frac{r}{5-r}$$ Solving the above equation we get $r=\frac{5}{2}$, but in that case $\beta=90^{\circ}$. Where I went wrong?


The strategy is fine, but you have made an algebraic mistake: $\frac{(5-2 r)+\sqrt{-2 r^2+25}}{\sqrt{2}(5-r)}$ is only $\cos \alpha + \sin \alpha$, not $\frac{\cos \alpha + \sin \alpha}{\sqrt 2}$. Solving $$\frac{(5-2 r)+\sqrt{-2 r^2+25}}{2(5-r)} = \frac{r}{5-r}$$ instead gives $r = \frac{20}{9}$.
A simpler approach is to use the two right triangles below:
The larger right triangle (with height $h$) has $h^2 + (5-r)^2 = (5+r)^2$, from which $h = \sqrt{20r}$. Then, the smaller right triangle has $(h-5)^2 + r^2 = (5-r)^2$, from which $r = \frac{20}{9}$.