Assuming that $a>b^2$ show that $$\sqrt{a-b\sqrt{a+b\sqrt{a-b\sqrt{a+...}}}}=\sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}$$ (corrected)
This problem listed in a contest-math preparation book with the tag Russian-IMO-Longlist 1999.
It is not in the problem statement, but I'm assuming that $a,b \in \Bbb R$ and $a,b>0$. I tried two routes, without much progress:
(a) developed $x=\sqrt{a-b\sqrt{a+bx}}$, as the solution $x$ must be a fixed point of $f(x)=\sqrt{a-b\sqrt{a+bx}}$, but the development does not appear promising (or the required algebra is beyond my capabilities).
(b) evaluated $f(x)=\sqrt{a-b\sqrt{a+bx}}$ on $x=\sqrt{a-\frac{3b^2}{2}}-\frac{b}{4}$ with the hope of getting $f(x)=\sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}$. Perhaps a naive approach.
Not great results from both approaches.
Hints and solutions are welcomed. Sorry if this is a duplicate.
Simplifying $x = \sqrt{a-b\sqrt{a+bx}}$ gives \begin{equation*} \frac{a^2}{b^3}-\frac{2 a x^2}{b^3}-\frac{a}{b}+\frac{x^4}{b^3}-x = \frac{1}{b^3}(x^2+bx+b^2-a)(x^2-bx-a) = 0, \end{equation*} so that \begin{equation*} x = -\frac{b}{2}\pm\sqrt{a-\frac{3b^2}{4}}\text{ or } x = \frac{b}{2}\pm\sqrt{a + \frac{b^2}{4}}. \end{equation*} The two possibilities with minus signs are negative (since $a>b^2$), which we reject since clearly $x>0$. Further, since $x < \sqrt{a}$, we also reject $x = \frac{b}{2} + \sqrt{a+\frac{b^2}{4}}$, leaving only $$x = \sqrt{a-\frac{3b^2}{4}}-\frac{b}{2}.$$