Given the functions $$ p(θ) = \frac{S}{2} × \frac{\cos\bigl(n × (θ - α)\bigr)}{n^2 - 1}\\ \begin{align} X(θ) = \cos(θ) × &\left(p(θ) + \frac{S}{2} + A\right) - \sin(θ) × p'(θ) - p(0)\\ Y(θ) = \sin(θ) × &\left(p(θ) + \frac{S}{2} + A\right) + \cos(θ) × p'(θ) - p\left(\frac{π}{2}\right) \end{align} $$ the curve $\bigl(X(t), Y(t)\bigr), 0° ≤ t < 360°$ is a smooth, regular $n$-sided polygonal Curve of Constant Width (CoCW) where
- $p'(θ)$ is the derivative of $p(θ)$ with respect to $θ$,
- $A > 0$ is the radius of the smallest osculating circle on the curve,
- $S + A > 0$ is the radius of the largest osculating circle on the curve,
- $S + 2 A$ is the total width of the curve, and
- $α$ is an angle parameter that rotates the curve about the origin.
- (If $A = 0$, the curve has sharp corners at the "vertices" and thus loses the property of being everywhere-smooth, but everything else holds. If $S = 0$, the curve becomes a circle with radius $A$.)
We also define $N := n - \sin\left(\frac{π}{2} × n\right)$.
As $α$ varies and the curve rotates, it stays bounded inside a regular polygon with $N$ sides centered on the origin with width and height equal to $S + 2 A$ (dashed white lines in the GIFs below). At all times, the curve touches every side of the bounding polygon at exactly one point (pink points in the GIFs below). However, the rotating curve doesn't completely cover the square; the covered area (solid white in the GIFs below) has rounded corners. I want to find the precise shape of those corners.
The bounding-box isn't completely rounded. It has a straight line-segment as the center of all four sides, with length $\frac{S × N}{n^2 - 1}$ (bounded by yellow lines in the GIFs above). However, I'm not sure how to find mathematical descriptions of the curves that join those straight segments together.
A Note on Osculating Circles
Given a point on a curve, an osculating circle shares a tangent with the curve at that point and has curvature (inverse of radius) equal to the curvature of the curve at that point. With the CoCW in this question being defined by the functions $X(t)$, $Y(t)$, we can draw two points on the curve $P_{1} = \bigl(X(β), Y(β)\bigr)$ and $P_{2} = \bigl(X(β + π), Y(β + π)\bigr)$. Because we are dealing with a CoCW, the tangents of the curve at these two points are parallel for any $β$ and the center-points of the osculating circles at both points are the same, located at $P_{0} = \bigl(X(β) - Y'(β), Y(β) + X'(β)\bigr)$. As we let $β$ vary, the smallest radius of the osculating circles on this curve will be $A$, and the largest radius will be $S + A$.
I have looked at this problem, reformulated it a bit, and developed a credible solution to envelope of curves described in the original post (OP). I work in the complex plane, so have reformatted the equations as follows,
$$ p(\theta)=\frac{S}{2}\frac{\cos (n(\theta-\alpha))}{n^2-1}\\ z(\theta)=\bigg(p(\theta)+ \frac{S}{2}+A \bigg)e^{i\theta}+i p’(\theta) e^{i\theta} $$
You’ll notice that I’ve dropped the translation terms $p(1)+ip(\pi/2)$ as they become superfluous when the solutions for various $\alpha$ are aligned in a single square box and the origin is moved to the center. Here, as in the OP, we have taken $S=1.25, A=S/2$.
To delineate the envelope of the solutions, i.e., the purple zone in the OP, I plotted the solution for 360 values of $\alpha$ on a single plot. This is shown in the first figure below. The red circle is shown for reference and comparison. The blue dashed lines indicate the straight sections of the envelope as called out in the OP. Our job is to find an analytic approximation to the outer envelope, The red star shows the pseudo-origin for developing the curve fit of the envelope. It is obvious that the envelope is not circular but must be close to it. Moreover, we see that it must be symmetric about the $45^{\circ}$-line.
This suggested to me to look at the superellipse. (Some readers may be familiar with the squircle, which will do just as well.) The superellipse is a special case of the superconics curves. The simple form for a curve in the first quadrant is
$$ y=(1-x^q)^{1/q},\quad x\in[0,1] $$
We note that $q=2$ is a circle, of course. And this must be scaled to the actual size of the envelope (details below). I found empirically the for $n=5$ as in the figure below, $q\approx2\cdot1.025641$. The envelope so calculated is shown in magenta in the figure.
Specifically, the pseudo-origin is given by $z_0=d(1+i)$, where, as per the OP,
$$ d=\frac{S}{2}\biggr( \frac{n-\sin(n\pi/2)}{n^2-1}\biggr) $$
The scaling factor is the distance from the origin to the side wall ($w=S/2+A$) less the distance to the pseudo-origin, thus giving the scaling factor $s=w-d$.
Similar calculations were carried out for the case $n=3$. Everything up to the empirical calculation of $q$ is identical, except for $n$, of course. Here we found that $q\approx2\cdot1.086957$. These results are shown in the second figure below.
We also looked at $n=7$ and $n=9$. Here we found that there are eight flat sections rather than four and did not go any further for the time being.