I have already shown the set $$\mathcal{B}=\{(x_{1},x_{2})\in \mathbb{R}^{2}: x_{2}\leq (x_{1})^{2}\}$$ to be non-convex, closed, and not bounded.
Now, I need to find the set of extreme points of $\mathcal{B}$ as well as its recession cone.
Definition of a Recession Cone: Let $X\subset \mathbb{R}^{m}$ be a convex set. The set $$X_{\infty}=\{d:X+d\subset X\}$$ is called the recession cone of $X$. (Sometimes also called the asymptotic cone.)
Definition of an Extreme Point: A point $x$ of a convex set $X$ is called an extreme point of $X$ if no other points $u,v\in X$ exist such that $$x=\frac{1}{2} u+ \frac{1}{2}v.$$
My problem is that both of these definitions specifically make mention of convex sets, while I am m working with a set that is not convex. I do know, however, that a set of discrete points consist solely of all extreme points while not being a convex set, so it is possible to find them in the case of $\mathcal{B}$; I just don't be know how.
I also don't know, in general how to find recession cones.
Could somebody please show me how to do this? I would be forever grateful and it would help me tackle similar problems.
Thank you.
Suppose $d$ is such that ${\cal B}+d \subset {\cal B}$.
Since $(x,x^2) \in {\cal B}$ we must have $x^2+d_2 \le (x+d_1)^2$ which simplifies to $d_2 \le 2 x d_1 + d_1^2$ for all $x$ hence $d_1 = 0$ and $d_2 \le 0$.
Suppose $d = (0, d_2)$ with $d_2 \le 0$, then is is easy to verify that ${\cal B}+d \subset {\cal B}$.
Hence ${\cal B}_\infty = \{ (0,t) | t \le 0 \}$. (Note that in general the set of recession directions is not necessarily a cone when the underlying set is not convex.)
Regarding extreme points:
Suppose $x \in {\cal B}$ and $x_2 < x_1^2$. The for sufficiently small $t>0$ we have $x_2+t < x_1^2$ and so $(x_1, x_2-t), (x_1, x_2+t) \in {\cal B}$. Since $(x_1,x_2) = {1 \over 2} (x_1, x_2-t) + {1 \over 2} (x_1, x_2+t)$ we see that $x$ is not extreme.
Now suppose $x \in {\cal B}$ and $x_2 = x_1^2$. If $x_1 = 0$ then we can see that $(-1,0), (1,0) \in {\cal B}$ and $(0,0) = {1 \over 2} (-1,0) + {1 \over 2} (1,0)$ so $(0,0)$ is not an extreme point. So we can suppose that $x_1 \neq 0$.
Since the curve $x \mapsto x^2$ is convex, we have that the line tangent to the curve lines below the curve, hence $(x, x_1^2+ 2 x_1(x-x_1) \in {\cal B}$ for all $x$ (this is straightforward to verify directly). In particular, if we choose $x = x_1 \pm 1$ we see that $(x_1,x_2) = {1 \over 2} (x_1, x_2-2 x_1) + {1 \over 2} (x_1, x_2-2 x_1)$ and so $(x_1,x_2)$ is not extreme.
(Note that the concept of extreme points doesn't really have much meaning when applied to non convex sets.)
Addendum: To show that $(x, x_1^2+ 2 x_1(x-x_1) \in {\cal B}$ for all $x$ we just need to show that $x_1^2+ 2 x_1(x-x_1) \le x^2$. This follows from the inequality $(x-x_1)^2 \ge 0$.