I'm trying to draw:
A set of $N$ (edit) irregular polygons one inside the other, where the innermost should be an equilateral triangle, enclosed by a square, enclosed by a pentagon, etc. Where all the points of each $N$-polygon touch sides of the $(N+1)$ - polygon. (Let's say up to $N = 8$)
Till now, I've began by drawing an equilateral triangle, taking one of the three vertices as reference point and I've tried to figure out (with no success) some relation between it and the next, encircling, polygon, such that the triangle vertices touch three of square's edges.
I can use some help in the form of any known function or recurrent relation related to the Cartesian coordinates of a polygon with N-vertices.
Edit: For the first two polygons, i.e. an equilateral triangle and a square, just noticed that the condition is fulfilled if the side of the triangle equals the side of the square. Graphically looks like this:
I wonder if this condition holds for the next iteration.
P.S. I'm using simple Euclidean Geometry to find the coordinates and some C++ (along with FLTK) to plot them, so the above plot is not just a picture.
Second Edit: as it turns out the only way the above condition is satisfied for the polygons to be irregular. Therefore, I'm adding it to the statement of the problem.
Let me explain why it is impossible to inscribe a regular pentagon into a regular hexagon in the way you want. Let's suppose that the side $AB$ of the hexagon contaning no pentagon vertex is parallel to side $GH$ of the pentagon (see picture). I'll take hexagon sides of unit length and set $x=AH$ so that $GH=1+x$ and $FH=1-x$. We can determine $x$ so that $HI=HG$ and $\angle GHI=108^\circ$: applying the sine rule to triangle $FHI$ one has $(1-x):\sin48^\circ=(1+x):\sin120^\circ$ so that $x$ is uniquely determined. By the same rule we can compute $FI=(1+x)\sin12^\circ/\sin120^\circ$ so that $FI$ is determined too. By simmetry the fourth vertex $K$ of the pentagon lies on $CD$ with $CK=FI$, while the fifth vertex $J$ must be the midpoint of $DE$. By the cosine rule applied to $EIJ$ one can then compute the length of $IJ$ and unfortunately it is not equal to $1+x$, so our pentagon is not regular.
One may think to circumvent this by choosing $HG$ not parallel to $AB$, but that is in fact useless: we have two parameters $AH$ and $BG$ to vary but we must satisfy the two conditions $HI=GH$ and $GK=GH$, so that the result is the same as before. And of course things get even worse for polygons having more sides (e.g inscribing a hexagon into a heptagon, and so on).
EDIT. If one drops the request of having regular polygons, than there are many ways to construct the requested sequence of nested polygons. One easy way is the following: start with any $N$-gon and choose at will $N-1$ of its sides: their midpoints can then be taken as vertices of an inscribed $(N-1)$-gon. The process can be repeated until a triangle is reached.