The question is so simple: how can I prove that $M$ (as defined below) is neither convex nor concave (in the real sense)?
The point here is to define a new notion of convexity, the complex convexity, in contrast with the real one; so this example should exhibits a surface $M$ for which, although it isn't convex/concave in the real sense, with this new notion we can say that it's complex convex.
The hypersurface is $M:=\partial\Omega$, where $\Omega:=\{(x_1,y_1,x_2,y_2)\in\Bbb R^4\;:\;y_1<-2x_2^2+y_2^2+\dots\}$ where the dots are terms in $z_2,\bar z_2$ (the book confuses deliberately $\Bbb C^2$ with $\Bbb R^4$ taking one or the other as he wants) of degree $\ge3$.
I try to study the real convexity of $M$ in various ways:
- With the definition of a convex function in several variables with $f(x_1,y_1,x_2,y_2)=y_1+2x_2^2-y_2^2-\dots$;
- Computing the determinant of the Gauss map of the parametrization of the surface (which is the curvature $K$, and we know that the surface is convex iff $K>0$).
But all seems to depend on the (unwritten) terms of degree $\ge3$.
How can I prove that $M$ is neither convex nor concave (in the real sense)?
First notice that point $(0,0,0,0)$ is a point on the boundary. If we can show that $M$ is not convex nor concave at this point - then we will be done. For ease of notation we write tuples as $(x_1,x_2,x_3,x_4)$. Notice that $\varphi(x) = x_2 +2x^2_3 - x_4^2 - \cdots$ is a defining function. Now calculating the Hessian matrix for $\varphi$ at $(0,0,0,0)$ we get $$\left[\frac{\partial \varphi}{\partial x_i \partial x_j}\right] = \left[ \begin{array}{ccc} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 4 & 0 \\ 0 & 0 & 0 & -2 \\ \end{array} \right] $$ To show convexity at $(0,0,0,0)$ fails we need to see that the Hessian matrix isn't positive semidefinite on the tangent space at $(0,0,0,0)$. Similarly to show concavity fails we need that the Hessian matrix isn't negative semidefinite on the tangent space. This is clear since the tangent space consists of $span (e_1,e_3,e_4)$.
For further details see "The Geometry of Domains in Space" page 192.