$f(x,y)=xy^2$; constraint $x^2/a^2 + y^2/b^2 = 1$
I took the gradients of $f(x,y)$ and $g(x,y)$. Used substitution method to find $x = \lambda(1/b^2)$ and $y = (\sqrt2\lambda/ab)$. However, I have got lost after this and would really appreciate any kind of help.
You are almost there. Plug in your $x$ and $y$ in the equation for the constraint.$$\frac{\lambda^2}{a^2b^4}+\frac{2\lambda^2}{a^2b^4}=1$$ You get $$\lambda^2=\frac13a^2b^4$$ Now find $x$ and $y$ from here. You should get$$x=\pm\frac1{\sqrt 3}a$$ and $$y=\pm\sqrt{\frac23}b$$ You need to calculate $f(x,y)$ at these four points:$$f(x,y)=\pm\frac2{3\sqrt3}ab^2$$