discuss convexity of the following set?

Question

discuss convexity of the following set ?

$$M= \{(x,y)∈\Bbb R^2 : x^2+y^2≥a^2 ,x^2+y^2≤b^2 ,x>0,y>0\} $$

Answer 1

For this to be a convex set, you need to be able to draw a line $L$ between any two points $(x_1, y_1), (x_2,y_2) \in M$ such that every point $(x,y) \in L$ is also in $M$.

In this case, it's obvious that the set $M$ can only be convex if $a = 0$ and $b>0$ (Naturally, $b>a$).

If $a>0$, there will always exist a small $\epsilon > 0$ such that a line $L$ from $(a,\epsilon), (\epsilon, a)$ will always pass through your "donut hole".

Convex sets are important in discussing the uniqueness of solutions in optimization problems --which is what I'm assuming you're studying given the tags you've put on this question. Consider reading the wiki on this topic, the images on the right sidebar illustrate my point: http://en.wikipedia.org/wiki/Convex_set

Edit: Typo(s)