I have a 2D polygon with known coordinates $ABCDEFG\,(x, y)$, and also some points inside and outside the shape, $K(x)$.
I want to figure out a formula, how to calculate the nearest point $C(x)$ and its coordinates $(x, y)$ from $K(x)$ on the shape. The polygon can be different, so the points so the formula needs to be generalized. Sadly I have no idea how to do this, so any help appreciated.


Let's assume that the $N$-sided polygon has vertices $(x_i, y_i)$ in clockwise or counterclockwise order, $i = 1 \dots N$, with $(x_0, y_0) = (x_N, y_N)$, and we wish to find the distance from point $(x_p, y_p)$ to the perimeter of the polygon.
As Théophile mentioned in a comment to the question, the simple (but perhaps not optimal) method is to find the minimum distance between the point and each edge line segment.
The way I prefer to find the distance between the line segment $(x_A, y_A) - (x_B, y_B)$ and point $(x_p, y_p)$, is parametrise the line segment with $0 \le t \le 1$, $$\left\lbrace \begin{aligned} x(t) & = (1 - t) x_A + t x_B \\ y(t) & = (1 - t) y_A + t y_B \\ \end{aligned} \right. \tag{1}\label{G1}$$ The line segment corresponds to $0 \le t \le 1$, so if $t \le 0$, the closest point to $(x_p, y_p)$ is $(x_A, y_A)$; if $t \ge 1$ the closest point to $(x_p, y_p)$ is $(x_B, y_B)$; otherwise, the closest point to $(x_p, y_p)$ is $(x(t), y(t))$.
There are many ways to find $t$ that corresponds to the minimum distance between $(x_p, y_p)$ and the line extending the edge line segment. Suffice it to say, they all yield a single solution, $$t = \frac{ ( x_p - x_A ) ( x_B - x_A ) + ( y_p - y_A ) ( y_B - y_A ) }{ (x_B - x_A)^2 + (y_B - y_A)^2 }$$
(If you precalculate $\vec{m}_i = \left(\frac{x_B - x_A}{(x_B-x_A)^2 + (y_B-y_A)^2} ,\, \frac{y_B - y_A}{(x_B-x_A)^2 + (y_B-y_A)^2}\right)$ for each edge $i$, you can calculate $t_i = (\vec{p} - \vec{v}_i) \cdot \vec{m}_i$ very efficiently, with just two scalar multiplications, two scalar subtractions, and one scalar addition per edge. To calculate the point when $0 \lt t_i \lt 1$, additional four scalar multiplications, two additions, and two subtractions are needed. The distance squared is another two scalar multiplications and an addition. Therefore, this way the worst case cost per edge is eight scalar multiplications and nine additions or multiplications – and that is quite acceptable and efficient, in computer geometry terms.)
Because the distance squared is a monotonic function of the distance, we don't need the square root for each edge line segment; we can find the minimum squared distance instead. This minimum distance $d_i^2$ for the edge line segment ending at vertex $i$ is $$d_i^2 = \begin{cases} (x_{i-1} - x_p)^2 + (y_{i-1} - y_p)^2, & t \le 0 \\ (x(t) - x_p)^2 + (y_(t) - y_p)^2, & 0 \lt t \lt 1 \\ (x_i - x_p)^2 + (y_i - y_p)^2, & 1 \le t \\ \end{cases} \tag{2}\label{G2}$$
Here is a CC0-licensed Python3 example program, that implements a Point class to describe 2D points and vecors, a Polygon class to describe closed 2D polygons, and when run, generates a random 10-sided polygon and ten random points, and saves
example.svg, an SVG image you can open in your browser, showing the polygon (in gray on white background), the random points (red dots), and blue lines from each random point to the closest point on the polygon perimeter.Save the following as
example.py, and run it using e.g.python3 example.py:The classes and methods have descriptions, so you can run
pydoc3 example(after saving the aboveexample.py) to see those.