For $a>b^2$, prove that $\sqrt{a-b\sqrt{a+b\sqrt{a-b\sqrt{a+\cdots}}}} = \sqrt{a-\dfrac34b^2}-\dfrac12b$.
Attempt: After assuming the value of the nested radicals to be $S$, I got $$S = \dfrac{\left(\dfrac{a-S^2}{b}\right)^2-a}{b},$$ but now I don't have any idea to solve $S$. The result must be the RHS and hence this will prove the equation above.
Regards.
The equation you got to can be rearranged to \begin{eqnarray*} S^4-2aS^2-b^3S+a^2-ab^2=0. \end{eqnarray*} Lucky for us this can be factorised to \begin{eqnarray*} (S^2+bS+b^2-a)(S^2-bS-a)=0. \end{eqnarray*} The first bracket will now give you the formula you require.