- I have a set: $$\{ (x, y, z) : z < x^{2} + y^{2} + 4xy \}$$
I want to prove that this set is convex or not. - I used general definition of convex set i.e.: $$z_{1}\alpha + (1-\alpha)z_{2} < (x_{1}\alpha + (1-\alpha)x_{2} + y \alpha + (1-\alpha)y_{2})^{2} + 2(...)(...)$$ But it looks too complex and I don't come with any solution.
- After that, I used another approach "It isn't convex set. For example: $A_{1} (0.1; 0.1; 0)$ and $A_{2} (-0.4 ; 0.1; 0)$. Both of this points in the set. But their linear combination with $\lambda = 0.5$ isn't in the set $A_{3} (-0.15; 0.1; 0)$". So, I choose two points and proved that their linear combination isn't in the set, thus it isn't convex set.
- But, I feel that it isn't a good idea, because I spend too much time for choosing "correct combination". Is there is a way to prove that it is/isn't convex set via general definition? I feel that something can be done with the first method.
Check some 'standard' points, such as the origin and notice that $(x,y,z)=(0,0,0)$ is not in the set.
Now see if there are points in the set on 'either side' such that the origin is on the line joining the points.
For example, take $(1,0,0), (-1,0,0)$ which are in the set and a convex combination ${ 1 \over 2 } ((1,0,0)+(-1,0,0))$ is not.