Lets say you imagine a circle with the radius $R$ and you inscribe a regular polygon with $n$ sides in it, whose side we know will then be: $$a=2R*sin(\frac{180}{n})$$
Then you draw a set of circles so that in each point of the polygon there is a circle which has the radius $R$. (Blue circles)
After that, you then draw another set of circles that pass through the intersections of the circles in the first set. (Red circles)
For $n=3$ It would look like this:
The green circle is twice the size of our imagined circle, and we also see that here there is only one red circle whose radius is the same as our imaginary circle in which the polygon is inscribed.
Let's start increasing our number of sides:
So, every two more sides we get a new circle, and all others increase in size with the outer one approaching the green circle. Lets set then $n$ to 32 for example:
Question
How can the the radius $r_m$ ($m=1,2,3...$) of the $m$th red circle be calculated for a polygon with $n$ sides inside the imagined circle of radius $R$? (Is there a Formula or expressions to be used?)
Thanks JeanMarie for answering this question: $$ r_m= 2R \ \cos{( \frac{m}{n}\pi )}$$
Use inversion (see def. below) with inversion circle your green circle. In the $n$th case, your circles will be transformed into a regular n-sided circumscribed polygon with a vertex $V$ at a distance (see figure) $\dfrac{10}{\cos\frac{\pi}{n}}$ from the origin.
The inverse $V'$ of point $V$ will thus be at a distance $10 \ \cos \frac{\pi}{n}$ from the origin.
Furthermore, $V$, being at the intersection of 2 sides, its image $V'$ will be at the intersection of the images of theses sides, i.e. the intersection of 2 circles.
The answer is thus that the radius is $10 \ \cos \frac{\pi}{n} \ \ \ (1)$, which tends to 10 when $n$ tends to $\infty$.
Definition: Inversion wrt to a circle with center $O$ and radius $r$ (here $r=10$) is the non linear transform such that the image of a point $P$ is $P'$ such that $O,P,P'$ aligned and $\vec{OP}.\vec{OP'}=r^2$.
See http://mathworld.wolfram.com/Inversion.html
**Edit **: (After a thorough remark of the author of the question, I rewrite this part). Let it be clear that formula (1) gives the radius of the circle that contains the outermost intersection points of circles that are internally tangent to the large circle (radius 10).
The penultimate level of intersection points is obtained by taking successively one out of two of these tangent circles, giving rise, by inversion to a different circumscribed polygon (hopefully non convex as in the figure we give) ; thus the same reasoning as before will end up with a radius $10 \ \cos \frac{2\pi}{n} \ \ \ (2)$. Then, we will take circles 3 by 3, and more generally $k$ by $k$, ...
Thus the general formula is
$$R_k=10 \ \cos \frac{k\pi}{n} \ \ \ \ k=1\cdots(n/2-1)$$
Remark: this formula explains the "3D interpretation" one can have of your last graphics as a view from above of a terrestrial globe with parallels at latitude $\frac{k\pi}{n}.$
Figure: The case $n=8=2^3$ with the 3 polygons having their vertices in correspondence, through inversion, with the interesctions of the families of circles.