I want to determine the extrema of $5x^2+7y^2-14xy$ on $D:=\left\{(x,y)\in \mathbb{R}^2:\max(x,y) \le 1\right\}$.
We didn't learn about Lagrange multipliers yet so I would like to solve this problem without them. It seems obvious that the extrema have to be at $x=y=0$ and $x=y=1$ but how can I reason this?
You don't need the Lagrange multipliers. All you need is to find the critical points, check the points on the boundary, as well as any singular points. Obviously there aren't any singular points and we first try to find the critical ones. Let $f(x,y) = 5x^2 + 7y^2 - 14xy$, then we have:
$$f_x = 10x - 14y \quad \quad f_y = 14y - 14x = 0$$
Equating them to zero we get that $14x = 14y = 10x$ and so $x= y= 0$
Now for the boundary first plug in $x=1$ and treat the function as a single variable function by letting $g(y) = f(1,y)$. Then we have:
$$g'(y) = 14y - 14$$
Equating it to zero we get that $y = 1$. Now do the same by setting $x=1$ and $h(x) = f(x,1)$ to get:
$$h'(x) = 10x - 14$$
This yields that $x=\frac{7}{5}$, however this isn't posible, as $x\le 1$. Thus you only need to check the points $(0,0)$ and $(1,1)$. Now to determine the nature of the $(0,0)$ you could apply the second derivative test and you'll get that:
$$H_f =\begin{vmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \\ \end{vmatrix} = \begin{vmatrix} 10 & -14 \\ -14 & 14 \\ \end{vmatrix} = - 56$$
As $H_f < 0$ we get that $(0,0)$ is a saddle point. To determine the nature of the point $(1,1)$ you need to apply the second derivative to $g(y)$. As $g''(y) = 14 > 0$ we get that at $(1,1)$ f attains a local minimum.