Let $f: \mathbb{R^2} \to \mathbb{R}$ with $f(x,y) = x^2 - y^2$.
We have to find all local and global extreme points of $f$ restricted on the set $L := \{(x,y) \in \mathbb{R^2} \mid x^2 + 4y^2 \leq 4 \}$. According to WolframAlpha, it's just an ellipse.
Since $f$ is continuous and $L$ is a compact set, the function $f$ has no global extreme points in $L$.
Therefore, we have to check all critical points and compare the function values.
The gradient of $f$ is given by $$\nabla f(x,y) = \begin{pmatrix} 2 x \\ -2y \end{pmatrix}$$
For $(x,y) \in L$ it holds that $\nabla f \neq 0$. Therefore, $f$ has no local extreme points in the within $L$.
So, we try to find extreme points on the boundary of $L (\partial L)$ with lagrange multiplicators.
Our side condition is $h(x,y) = x^2 + 4y^2 - 4$.
We get $J h'(x,y) = J (2x, 8x) = 4$
The Lagrange function is
$$F(x,y,\lambda) = x^2 - y^2 + \lambda \cdot (x^2 + 4y^2 - 4)$$
We are looking for $(x,y,\lambda)$ with
$$\text{(1) } F_x(x,y,\lambda) = 2x + 2 \lambda x = 2x (1 + \lambda) = 0$$ $$\text{(2) } F_y(x,y,\lambda) = -2y + 8 \lambda y = 2y (-1 + 4 \lambda) = 0$$ $$\text{(3) } F_\lambda(x,y,\lambda) = x^2 + 4 y^2 -4 = 0$$
Now I'd have to look at $3$ cases:
Case $1$: $x \neq 0$
Case $2$: $x = 0$ and $y \neq 0$
Case $3$: $x = y = 0$
But I'm not sure how to continue from here. I don't know how to get the critical points if I look at each case. I know that I have to put in the value of the critical point in the original function $f$, but how can I get these values?
You got the system$$\left\{\begin{array}{l}2x(1+\lambda)=0\\2y(-1+4\lambda)=0\\x^2+4y^2=4.\end{array}\right.$$Now:
So, the points that you get are $(\pm2,0)$ and $(0,\pm1)$. Now, use the fact that $f(\pm2,0)=4$ and that $f(0,\pm1)=-1$.