Let $f(x,y) = xy^2$ and the domain $D = \lbrace (x,y)| x,y\geq0, x^2 + y^2 \leq 3 \rbrace$
$f_x(x,y) = y^2$ and $f_y(x,y) = 2xy$
Therefore, the critical points should be $\lbrace (x,y)| y = 0, \sqrt{3} \geq x \geq 0 \rbrace$.
The determinant of the Hessian is $$\det(Hf(x,y))= \begin{vmatrix} 0 & 2y \\ 2y & 2x \end{vmatrix} = 0-4y^2$$
But this doesn't make sense to me because, if $y = 0$, then $f(x,0) = 0$. But on this interval, suppose I chose $(x,y) = (1,\sqrt{2})$, this would be larger and would be the maximum value of the system.
Why didn't finding the first partials and the Hessian allow me to compute the critical point?
Critical points are candidates for the max and min values.
Boundary points are also candidates.
But for the specified domain $D$, the only possible critical points are the points where $y=0$, and those points are on the boundary, hence are not actually critical points.
It follows that the maximum and minimum values of $xy^2$ must be on the boundary.
For the minimum, it's obvious that the minimum value is $0$, which occurs at all points on the boundary where $x=0\;$or $y=0$.
For the maximum, it's obvious that the maximum is positive, hence the only candidates are boundary points where $x,y > 0.\;$It follows that the maximum occurs on the circular part of the boundary, excluding the two endpoints.
Thus, letting $(x,y)=\bigl(\sqrt{3}\cos(t),\sqrt{3}\sin(t)\bigr)$, we want to maximize $\bigl(\sqrt{3}\cos(t)\bigr)\bigl(3\sin^2(t)\bigr)$, for $0 < t < {\large{\frac{\pi}{2}}}.\;$Then letting $g(t)=\bigl(\sqrt{3}\cos(t)\bigr)\bigl(3\sin^2(t)\bigr)$ and setting $g'(t)=0$, we get only one critical point for $0 < t < {\large{\frac{\pi}{2}}}$, namely $t=\cos^{-1}\left({\large{\frac{1}{\sqrt{3}}}}\right)$, which yields the maximum value for $f$ of $2$.
Alternatively, now that I see saulspatz's comment, since the circle has the equation $x^2+y^2=3$, maximizing $xy^2$ on the circular arc of the boundary, excluding the endpoints, is the same as maximizing the function $x(3-x^2)$ on the interval $0 < x < \sqrt{3}$. Taking the derivative and setting it to $0$, we get only one critical in the interval $0 < x < \sqrt{3}$, namely $x=1$, which yields the maximum value for $f$ of $2$.