To find the minimum and maximum of $f(x,y)=x^2y+2xy+12y^2$ in the domain $g(x,y)=x^2+2x+16y^2\leq8$, I'll use Lagragian multiplier's method.
$f(x,y)=x^2y+2xy+12y^2$
$\nabla f = \lambda \nabla g$
$(2xy+2y)\hat i +(x^2+2x+24y)\hat j = \lambda\left( (2x+2)\hat i + 32y\hat j\right)$
$(x+1)y = \lambda(x+1)$
$(y-\lambda) (x+1) = 0\tag 1$
$x^2+2x+24y = 32\lambda y\tag 2$
From (1), it is either $x = -1$ or $\lambda = y$
If $x = -1$
I plug the value of $x$ in $g(x,y)$ to find $y =\pm \frac{3}{4}$
Wolfram Alpha says $(-1,-3/4)$ is a maximum, so that's OK, my problem comes when $\lambda = y$. If $\lambda = y$, $$x^2+2x+24y-32y^2=0$$ $$\longrightarrow{y=\frac{1}{8}(\sqrt{2x^2+4x+9}+\frac{3}{8}}$$ $$\longrightarrow{y=\frac{3}{8}-\frac{1}{8}(\sqrt{2x^2+4x+9}}$$
But that does not help me in any way (I think). What can I do to find the minimum?
You can think that the extremes are either within the region or on the boundary. On the boundary you are solving the problem:
$$ \min/\max:~~ f(x,y)\\ \text{st}:~~~ g(x,y) - 8 =0 $$
As you point out this requires to solve the equations
\begin{eqnarray} \nabla f(x,y) &=& \lambda\nabla[g(x,y) - 8] \\ g(x,y) - 8 &=& 0 \end{eqnarray}
which is equivalent to
\begin{eqnarray} (1+x)(y-\lambda) &=& 0 \tag{1}\\ x^2 + 2x + 24y &=& 32 \lambda y \tag{2}\\ x^2 + 2x + 16 y^2 &=& 8 \tag{3} \end{eqnarray}
in the case $y\lambda$, eqns (2) and (3) become
\begin{eqnarray} x^2 + 2x + 24y &=& 32y^2 \\ x^2 + 2x + 16y^2 &=& 8 \end{eqnarray}
if you add them up you get
$$ 24y - 16y^2 = 32y^2 - 8 ~~~\Rightarrow~~~ y = \frac{1}{12} (3 \pm \sqrt{33}) $$
From here you can calculate $x$