Let $A = \{(x_1,x_2, x_3) \in \mathbb{R}^3 : x_1^2 + x_2^2 + x_3^2 \leq 1\}$ and $f : A \to \mathbb{R}$ be defined by $f(x_1,x_2,x_3) := x_1^2 + x_2^3 - 2x_1x_2 - x_3^2$ for $(x_1,x_2,x_3) \in A.$ By the EVT and continuity of $f, f$ must attain a global maximum. Prove that all points of global maximum are on the unit sphere $S = \{(x_1,x_2,x_3) \in \mathbb{R}^3 : \sum_{i=1}^3 x_i^2 = 1\}$.
I know that points of local maximum are critical points. We have that $\nabla f(x_1,x_2,x_3)$ is $(2x_1 -2x_2, 3x_2^2 - 2x_1, -2x_3).$ This is zero iff $(x_1, x_2, x_3) = (0,0,0)$ or $(0,\frac{2}3, \frac{2}3),$ but neither of these points are global maxima for $f$ (we easily have $f(1,0,0) = f(0,1,0) = 1$, which is greater than the value of $f$ at both of these points). By the second derivative test, $f$ attains a local maximum at a point where the Hessian of $f$ at $\vec{a} \in A$, whose $(i,j)$ entry is $\partial_i \partial_j f(\vec{a})$, is negative semi-definite. We have that the Hessian of $f$ is given by $Hf(\vec{a}) = \begin{bmatrix}2 & -2 & 0\\ -2 & 6a_2 & 0 \\ 0 & 0 & -2\end{bmatrix}\,\forall \vec{a} = (a_1, a_2, a_3).$ For a column vector $\vec{z} = (a,b,c)\in\mathbb{R}^3, z^T Hf(\vec{a})z = (2a-2b)a + (-2a + 6a_2 b) -2c^2.$ I'm not sure how to use this to determine local maxima though.
Let's define $\mathring{A}:=\{(x_1,x_2,x_3)\mid x_1^2+x_2^2+x_3^2<1\}$ and $cl(A):=\{(x_1,x_2,x_3)\mid x_1^2+x_2^2+x_3^2=1\}$, so $A=\mathring{A}\cup cl(A)$.
We can find all possible points $(x_1,x_2,x_3)\in \mathring{A}$ which might attain a global maximum by setting $Df(x_1,x_2,x_3)\overset{!}{=}0$ (first order condidtion aka F.O.C.) and then verifying that those points indeed lie in $\mathring{A}$. This is what you have already done and as you have pointed out those points are no global maxima nor do they lie on the unit sphere. As the function is continuous and defined on a compact set it must attain a global maximum (and a global minimum). We can easily find points $(x,y,z)$ and $(a,b,c)$ on $cl(A)$ which satisfy $f(x,y,z)>f(0,0,0)$ and $f(a,b,c)<f(0,\frac{2}{3},\frac{2}{3})$. So the global extrema must be attained by points in $cl(A)$. Hence, they lie on the unit sphere.