Start with a circle that has a regular polygon inscribed inside of it. Place it on top of a line so that one of the polygon's vertices is touching the line. Then imagine the circle rolling on the line until the next vertice is touching.
What is the area of the region between the line and rotated edge of the polygon, the region not passed over by the edge? What if a 2-gon, the diameter of the circle?

See the diagram and note the variable angle $\theta$, which we will parametrise the problem by.
From the diagram, note that the vertical distance from the closest edge of the square to the line is $y$. $x$ is the horizontal distance travelled along the line by the circular edge (this is also the arc length along the circumference of the circle). It is clear that you want to find the relationship between $y$ and $x$ and then perform the integration to find the required area under that curve.
By applying sine rule to the triangle in the diagram with two sides $R-y$ and $R$ and two angles $\theta$ and $\displaystyle \frac{\pi}{4}$ (and therefore other angle $\displaystyle \pi - (\frac{\pi}{4} + \theta) = \frac{3\pi}{4} - \theta$), you can write:
$\displaystyle \frac{R-y}{\sin \frac{\pi}{4}} = \frac{R}{\sin(\frac{3\pi}{4} - \theta)}$
Solving and simplifying, $\displaystyle y = R\left(1 - \frac 1{\sin\theta + \cos\theta}\right)$ and $x = R\theta$.
From this, you can easily get: $\displaystyle y = R\left(1 - \frac 1{\sin\frac xR + \cos\frac xR}\right)$
The indefinite integral $\displaystyle \int ydx$ is rather tedious, but a closed form indefinite integral exists. You can find it at Wolfram link
The bounds you require for the definite integral to get the area are $x = 0$ and $\displaystyle x = \frac{R\pi}{2}$ (as the circle rolls so that the next vertex touches the line, it's gone through a quarter of its circumference). That gives the value of area $\displaystyle A = \frac 12R^2\left(\pi - 2\sqrt 2 \tanh^{-1}\left(\frac 1{\sqrt 2}\right)\right) \approx 0.324R^2$. I'm not sure how to copy paste the link to the original result.