Let $R_1$ be regular $n$-sided polygon on the plane (square, pentagon, hexagon, etc).
Now from this position we start to rotate this polygon about its center of gravity obtaining figure $R_2$.
- How to calculate the angle of rotation $\alpha$ for the case where common area of $R_1$ and $R_2$ i.e. area ($R_1 \cap R_2)$ will be minimal? (intuition tells what possible solution could be but how to prove it?)
- Does some simple method exist for solution of this problem in general case? (preferably with the use of rotations matrices)
- The procedure for $n$-odd and $n$-even could be the same or we should to differentiate between these two cases?
Additionally:
- Could it be proven that the shape obtained for situation of minimal area is also a regular polygon ( $2n$ sided) as we see in the below picture of pentagon made by Joseph?


The minimum is achieved at $\alpha = \frac{\pi}{n}$ and at minimum, $R_1 \cap R_2$ is a regular $2n$-gon.
Choose a coordinate system so that $R_1$ is centered at origin and one of its vertices lies on $x$-axis.
Let $\rho(\theta)$ be the function which allow us to parametrize $\partial R_1$ in following manner:
$$\mathbb{R} \ni \theta \quad\mapsto\quad (x,y) = (\sqrt{2\rho(\theta)}\cos\theta,\sqrt{2\rho(\theta)}\sin\theta) \in \partial R_1$$
In terms of $\rho(\theta)$, we have
$$f(\alpha) \stackrel{def}{=} \verb/Area/(R_1 \cap R_2) = \int_0^{2\pi} \min(\rho(\theta),\rho(\theta-\alpha)) d\theta$$
Since $R_1$ is a regular $n$-gon and one of its vertices lies on $x$-axis, $\rho(\theta)$ is even and periodic with period $\frac{2\pi}{n}$. In fact, it strictly decreases on $[0,\frac{\pi}{n}]$ and strictly increases on $[\frac{\pi}{n},\frac{2\pi}{n}]$.
As a result of these, $f(\alpha)$ is even and periodic with same period. To determine the minimum of $f(\alpha)$, we only need to study the case where $\alpha \in \left[0,\frac{\pi}{n}\right]$.
For $\alpha \in \left[0,\frac{\pi}{n}\right]$ and $\theta \in \left[0,\frac{2\pi}{n}\right]$, the curve $\rho(\theta)$ and $\rho(\theta - \alpha)$ intersect at $\frac{\alpha}{2}$ and $\frac{\alpha}{2} + \frac{\pi}{n}$.
This leads to $$\begin{align}f(\alpha) &= n\left[ \int_{\frac{\alpha}{2}}^{\frac{\alpha}{2}+\frac{\pi}{n}} \rho(\theta) d\theta + \left( \int_0^{\frac{\alpha}{2}} + \int_{\frac{\alpha}{2}+\frac{\pi}{n}}^{\frac{2\pi}{n}} \right)\rho(\theta-\alpha)d\theta \right] = 2n\int_{\frac{\alpha}{2}}^{\frac{\alpha}{2}+\frac{\pi}{n}} \rho(\theta) d\theta\\ \implies \frac{df(\alpha)}{d\alpha} &= n\left(\rho\left(\frac{\alpha}{2}+\frac{\pi}{n}\right) - \rho\left(\frac{\alpha}{2}\right)\right) \end{align} $$ At the minimum, we have $$\frac{df(\alpha)}{d\alpha} = 0 \implies \rho\left(\frac{\alpha}{2}\right) = \rho\left(\frac{\alpha}{2} + \frac{\pi}{n}\right) = \rho\left(\frac{\alpha}{2} - \frac{\pi}{n}\right) = \rho\left(\frac{\pi}{n} - \frac{\alpha}{2}\right) $$ But $\frac{\pi}{n} - \frac{\alpha}{2}$ also belongs to $[0,\frac{\pi}{n}]$ and $\rho(\theta)$ is strictly decreasing there, this means
$$\frac{\alpha}{2} = \frac{\pi}{n} - \frac{\alpha}{2}\quad\implies\quad \alpha = \frac{\pi}{n}$$
Please note that this argument doesn't use the explicit form of regular $n$-gon. It uses
This means the same argument should work for other shapes with similar properties. e.g. those obtain from filling the "interior" of a regular star polygon.