Study the absolute minima and maxima of $$f(x,y)=(x^2-y^2)(x-2)$$ in the triangle $A$ of this vertices: $$O=(0,0) \qquad P=(2,2) \qquad Q=(2,-2)$$
I consider the set: $$A=\{ (x,y) \in \mathbb{R}^2 : 0 \le x \le 2 \ , \ -x \le y \le x \}$$
I have tried to find minima and maxima in $int(A)$
Partial derivatives:
$$f_x(x,y)=3x^2-y^2-4x \\ f_y(x,y)=-2xy+4y \\ f_{xx}(x,y)=6x-4 \\ f_{yy}(x,y)=-2x+4 \\ f_{xy}(x,y)=f_{yx}(x,y)=-2y$$
Hessian matrix:
$$H(x,y)=\begin{bmatrix}6x-4 & -2y \\ -2y & -2x+4 \end{bmatrix}$$
Stationary points:
\begin{cases} 3x^2-y^2-4x=0 \\ -2xy+4y=0 \\ \end{cases}
$S_1=(0,0) \qquad S_2=(2,2) \qquad S_3=(2,-2)$
So $$S_1=O \qquad S_2=P \qquad S_3=Q $$
$$\det H(O)=\det H(P)=\det H(Q)=-16<0$$
$O,P,Q$ are saddle points
I try to search maxima and minima on the sides of triangle
$$y=x \qquad 0\le x\le 2 \\ F(x,x)=0 \qquad \forall x \in \mathbb{R}$$
$$y=-x \qquad 0\le x\le 2 \\ F(x,-x)=0 \qquad \forall x \in \mathbb{R}$$
$$x=2 \qquad -2\le y\le 2 \\ F(2,y)=0 \qquad \forall y \in \mathbb{R}$$
Is there any mistake?
Thanks!
A few items of feedback:
You missed a second critical point when $y=0$.
It is not enough to just look at the Hessian to determine the character of a critical point on the region boundary. Consider for instance $x^2-y^2$; there is a saddle point at the origin if this function is defined on the entire plane, but the origin is a minimum if the domain is, say, the cone $x \geq 0; -x \leq y \leq x$.