Let's say I have a circle with 1 point (cx, cy) on this circle (whatever the radius - I want to use cos() and sin() to solve this problem so the radius shouldn't matter).
I have the coordinates (x1, y1) (x2, y2) (x3, y3) (x4, y4) of those 4 points which are on a square (and I've just added my "radians" notes).
The center is (0, 0).
I want to know, using (cx, cy) where it's the closest between those segments: (point 1 and 2), (point 2 and 3), (point 3 and 4) or (point 4 and 1).
How would you do?
┌───────────────────────┬───────────────────────┐
│ 4 │ 1 │
│ │ / │
│ PI + (PI/2) / │
│ │ / │
│ │ PI + PI/2 + PI/4 │
│ │ or │
│ │ 2PI - PI/4 │
│ │ / │
│ │ / │
│ │ / │
│ │ / │
│ center │ │
├─── PI ────────────────┼─────── 0 / or 2 PI ───┤
│ │ │
│ │ │
│ / │ │
│ / │ │
│ / │ │
│ / │ │
│ / │ │
│ / │ │
│ PI/2 + PI/4 PI/2 │
│ / │ │
│ / │ │
│ 3 │ 2 │
└───────────────────────┴───────────────────────┘
You can take advantage of the fact that the square is axis-aligned to minimize the necessary computation. For example, the distance from the point $(cx,cy)$ to the vertical line $x = x_0$ is merely $d = |cx - x_0|$. However, you are considering just the segment of the line $x = x_0$ between, say $y_{\text{min}}$ and $y_{\text{max}}$. Well, if $y_\text{min} \leq cy \leq y_\text{max}$ then the distance from $(cx,cy)$ to the line segment is again $d = |cx - x_0|$. But if $cy > y_\text{max}$, then the distance from the point to the line segment is now the same as the distance from $(cx,cy)$ to $(x_0, y_\text{max})$. If $cy< y_\text{min}$, then the distance is the distance from $(cx,cy)$ to $(x_0, y_\text{min})$.
This is all true up to swapping $x$ and $y$ when considering the horizontal lines of the form $y = y_0$, so simply repeat this three-case algorithm for each line segment and take the least distance as the final answer.
Another thing you can do to reduce unnecessary calculation is to consider the two parallel sides at once. For instance, if the two sides are given by the lines $x = x_0$ and $x = x_1$, then the point $(cx,cy)$ is closer to the $x_0$ side if $|cx - x_0| < |cx - x_1|$, and vice versa. You can do this before checking if $y_\min \leq cy \leq y_\max$ to skip computing the distance from $(cx,cy)$ to the vertices which it couldn't possibly be closer to.