I found this def in my textbook: Let A be a subset of a Euclidean vector space X. Then
$$A^* = \{y \in X: (x|y) =\lt 1, \;\;\forall x \in A\}$$ is called to be an "extreme set" of A. (I'm not sure if I translate this right). Can you explain me what is (x|y)? Can't find it anywhere on Google. Thanks!
From the definition of extreme set at MathWorld, I would guess it means that on the line between $\mathbf x$ and $\mathbf y$:
$$(1-t)\mathbf x + t\mathbf y \in A$$
then $t\in[0,1]$.