The domain Is defined by the half unit circle: $\left \{ \left ( x,y \right )\in \mathbb{R}^2: x^2+y^2\leq 1, x\geq 0 \right \}$
for the function $f:D\rightarrow \mathbb{R}$. The function is $f(x,y)=2xy^2-2x^2$. I have shown that a min/max must exists since the function is continuous and the domain is both bounded and closed. I do not know how to proceed to find these points where the function is equal to its min/max. I tried with the Lagrange multiplier but it is just too messy for me to do. How would one do it directly? I found that the gradient vanishes at $(0,0)$

Since $x\geqslant 0, f(x,y) \leqslant 2x(1-x^2)-2x^2$. Now find $x$ between 0 and 1 to maximize this.
Finding the minimum is a lot easier since $2xy^2\geqslant 0$ so we must have $y=0$. Can you do the rest?
===============
We can also use polar coordinates. Let $x=R\cos \alpha, y=R\sin \alpha$, with $0\leqslant R \leqslant 1, -\frac{\pi}{2} \leqslant \alpha \leqslant \frac{\pi}{2}$.
(1) On boundary $S_1 = \left\{ x^2+y^2=1, x\geqslant 0 \right\}$ we have $R^2=1$ so $f(x,y) = 2\cos \alpha \sin^2 \alpha - 2 \cos^2 \alpha = 2\cos \alpha - 2\cos^3 \alpha - 2 \cos^2 \alpha$.
Now $\frac{\partial f}{\partial \alpha}= -2 \sin \alpha-6\cos^2\alpha (-\sin \alpha)-4 \cos \alpha (-\sin \alpha) \\ = 2 \sin \alpha (-1+3 \cos^2\alpha +2\cos \alpha)\\ = 2\sin\alpha (3\cos \alpha -1) (\cos \alpha+1) $
$\frac{\partial{f}}{\alpha}=0 \Rightarrow \sin \alpha =0 $ or $\cos \alpha = \frac{1}{3}$
When $R=1, \sin \alpha =0, x=1, y=0, f(x,y) = -1$.
When $R=1, \cos \alpha = \frac{1}{3}, x=\frac{1}{3}, y=\pm \frac{2\sqrt{2}}{3}, f(x,y)=\frac{10}{27}.$
(2) On boundary $S_2 = \left\{ x=0, -1\leqslant y \leqslant 1\right\}$ we have $f(x,y)=0$.
Therefore the minimum is $-1$ at $(1,0)$ and the maximum is $\frac{10}{27}$ at $(\frac{1}{3}, \pm \frac{2\sqrt{2}}{3})$