$ Let\;f \in\;\mathcal{C^3(U)\cup C(\bar U)}\;when\;\mathcal U\;is\;an\;open\;and\;bounded\;subset\;of\;\mathbb R^2.\; Assume\;that\;det\mathbf D^2f(x,y)\le 0\; for\; every\;(x,y) \in \mathcal U.$
$Prove\;that\; \max_{\bar U} f\;=\;\max_{boundary\;of\;U} f\; and\; \min_{\bar U} f\;=\;\min_{\mathcal boundary\;of\;U} f\;$
$By\; \mathbf D^2f(x,y) \; we\; denote\; \mathbf D^2f(x,y)\;=\;$$\begin{pmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \\ \end{pmatrix}\\$$ \\$ $where \;f_{xx} \; f_{xy}\; f_{yx}\; f_{yy} \; are\; partial\; derivatives\;of\; f $
I tried to prove this by setting $g_\varepsilon (x,y)=f(x,y)+ 2\varepsilon xy\;for\;\varepsilon \gt 0\;and\;\forall (x,y) \in U.$ Then by computing we get $det\mathbf D^2{g_\varepsilon (x,y)}=det\begin{pmatrix} g_\varepsilon{xx} & g_\varepsilon{xy} \\ g_\varepsilon{yx} & g_\varepsilon{yy} \\ \end{pmatrix}=det\begin{pmatrix} f_{xx} & f_{xy}+2\varepsilon \\ f_{yx}+2\varepsilon & f_{yy} \\ \end{pmatrix}=det \mathbf D^2 f(x,y)-4\varepsilon f_{xy}-4\varepsilon^2 \lt 0$ (Note that since $f_{xx}f_{yy}-f_{xy}^2 \le 0$ it follows that $f_{xy} \ge \sqrt{\vert f_{xx}f_{yy}\vert }\gt 0$)
Now, since $g_\varepsilon \in \mathcal C(\bar U)$ it attains its maximum at some point $(x,y) \in \bar U$. If $(x,y) \in U$ I derive a contradiction as follows. At a maximum it is necessary that $\mathbf Dg_\varepsilon (x,y)=0$ and $\mathbf D^2{g_\varepsilon (x,y)}$ be negative semi-definite. Thus $\max_{\bar U} g_\varepsilon = \max_{boundary\;of\; U} g_\varepsilon$.
Let $r \gt 0 $ such that $xy \le r^2\;\;\;\forall (x,y) \in U$
$max_{\bar U}f \le \max_{\bar U} g_\varepsilon = \max_{boundary\;of\; U} g_\varepsilon \le \max_{boundary\;of\;U} f + 2\varepsilon r^2$ and by letting $\varepsilon \to 0$ we deduce $\;\;\max_{\bar U} f \le \max_{boundary\;of\;U} f$
However I'm not pretty sure that actually exists such $r$ in $\mathcal U$. Could somebody improve this proof? I would appreciate any other help or hint because I've been stuck for days. Thanks in advance!
Here's a hit to get you started. If $g: \mathbb{R}^2 \to \mathbb{R}$ is $C^2$ and has a local maximum at $x\in \mathbb{R}^2$, then $D^2 g(x) \le 0$, i.e. $D^2 g(x)$ is negative semidefinite.
EDIT: (I typed the above while you were editing, so it looks like you know this already)
OK, you've got the above observation. The next important step is to realize that a positive semidefinite $2 \times 2$ matrix must have non-negative determinant, and a negative semidefinite $2 \times 2$ matrix must also have a non-negative determinant. One possible proof of this is to look at the eigenvalues, which must both have the same sign.
Now, this means that you actually want to show that your perturbation $g_\epsilon$ is such that $\det D^2 g_\epsilon < 0$. In your choice of $g_\epsilon$ you can actually take $\epsilon < 0$ if need be. You can do this based on the sign of $f_{xx} + f_{yy}$ at the point $(x_0,y_0)$ in order to make $\det D^2 g_\epsilon(x_0,y_0) < 0$.