Consider the function $$f(x) = (x_1-1)^2x_2$$
Consider the points of the form $\overline{x} = (1,x_2)$
a) Analyze the optimialty of first and second order for these points.
b) What we can say about $\overline{x}$ using these informations?
c) Use the expression of the function to obtain more conclusive informations about the characteristics of $\overline{x}$
a)
$$\frac{\partial f}{\partial x_1} = 2(x_1-1)x_2$$ $$\frac{\partial f}{\partial x_2} = (x_1-1)^2$$
$$\nabla f(1,x_2) = 0$$
$$\frac{\partial^2 f}{\partial x_1^2} = 2x_2$$
$$\frac{\partial^2 f}{\partial x_2^2} = 0$$
So the hessian at $(1,x_2)$ is
$$H(1,x_2) = \begin{bmatrix} x_2 & 1 \\ 1 & 0 \end{bmatrix}$$
$$\begin{bmatrix} a & b \end{bmatrix}\begin{bmatrix} x_2 & 1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix} a \\ b \end{bmatrix} = a^2 x_2^2 + 2ab$$
b) this matrix is not definite positive or definite negative, so I cannot say that the points of the form $(1,x_2)$ are extreme points but I also cannot say they are not.
c) I think that this item is asking me to use the function to prove $(1,x_2)$ are not extreme points. Let's analyze $(1,x_2) + \lambda (d_1,d_2)$
$$f(x) = (1+\lambda d_1 -1)^2 (x_2+\lambda d_2) = \lambda^2 d_1^2 (x_2+\lambda d_2)$$
But small variations on $\lambda$ do not change the signal of $f$
$$\frac{\partial^2 f(x_1, x_2)}{\partial x_1 \partial x_2}=2(x_1-1)$$
$$\frac{\partial^2 f(1, x_2)}{\partial x_1 \partial x_2}=0$$
$$H(1,x_2) = \begin{bmatrix} 2x_2 & 0 \\ 0 & 0 \end{bmatrix}$$
Same conclusion as you for the front part.
Now to analyze its optimality.
Consider $x_2 > 0$. Then, we can draw a ball around $(1,x_2)$ such that the $y$-coordinate is always positive. Hence $f$ is nonnegative on that ball, hence it is a local minimum.
Similarly for $x_2<0$, we can draw a ball such that the $y$-coordinate is always negative, hence $f$ is nonpositive on that neighborhood. Hence it is a local maximum.
For $(1,0)$, whenever we draw a ball, we can always find $x_1 \ne 0$ and it is possible for $x_2$ to be positive or negative, hence it is neither local maximum nor minimum.