($x^2+(y+1)^2=R^2$ should say $x^2+(y-1)^2=R^2$)
I am trying to derive a formula for the radius of the circle that is externally tangent to the internally tangent circles of the quarter-circle, and that is internally tangent to the quarter-circle.
Comment: this question is from a mathematics book, and it states that the radius is $\frac{4-2\sqrt{2}}{6-\sqrt{2}}R$
You should mean that $$x^2+y^2=(2R)^2\tag1$$ $$(x-\color{red}{R})^2+y^2=R^2\tag2$$ $$x^2+(y\color{red}{-R})^2=R^2\tag3$$ $$(x-a)^2+(y-b)^2=c^2\tag4$$
From $(1)(4)$, $$\sqrt{a^2+b^2}=2R-c\tag5$$ From $(2)(4)$, $$\sqrt{(a-R)^2+b^2}=c+R\tag6$$ From $(3)(4)$, $$\sqrt{a^2+(b-R)^2}=c+R\tag7$$
Since $a=b$ from $(6)(7)$, we have from $(5)(6)$, $$a\sqrt 2=2R-c\quad\text{and}\quad \sqrt{(a-R)^2+a^2}=c+R$$ Solving this system gives $$c=\frac{4-2\sqrt 2}{6-\sqrt 2}R.$$