The question is:
Consider the domain $\{ (x,y) \in \mathbb{R}^2 ; x \le y \}$ The function $h$ is mapped from this domain to $R$ — the set of all real numbers: $$h(x,y) = (x-2)^4 + (y-1)^4.$$ Find the minimum value of this function.
I found out the necessary partial derivatives — $h_x$ and $h_y$ and equated them to $0$ to get $x=2$ and $y=1$. Further, $h_{xx}$,$h_{yy}$ and $h_{xy}$ are all zero for these values of $x=2$ and $y=1$. So,H = $h_{xx}h_{yy} - h_{xy}^2$ is coming out to equal 0. How should I proceed now?
Note that the critical point $(2,1)$ is NOT in the given domain $D=\{ (x,y) \in \mathbb{R}^2 ; x \le y \}$. It follows that the minimal value of $h$ is not in the interior of $D$.
Now, in order to find the minimal value of $h$, you should consider the function $h$ restricted to the boundary of $D$, i.e. along the line $y=x$: $$f(x):=h(x,x)=(x-2)^4 + (x-1)^4$$ where $x\in \mathbb{R}$. We take the derivative of $f$ and find its critical points: $$f'(x)=4(x-2)^3 + 4(x-1)^3=4(2x-3)\underbrace{(x^2-3x+3)}_{>0}.$$ Therefore $x=3/2$ is the unique minimum point for $f$ and $(3/2,3/2)$ is the minimum point for $h$ with minimal value $h(3/2,3/2)=1/8$.