What is the mathematical technique, or a branch in mathematics that represents fitting a line on to a set of points such that it becomes a polygon?

Question

And what if the line that being fitted has an additional constraint of avoiding certain points while making the polygon?

Is there some specific fields I need to be exploring? For instance, set theory. The points that forming a polygon can be a set, and points under constraint condition can be another set. Another thinking direction was convex optimization of set points with constraints.

PS: I do not mean fitting of line as in machine learning.