Good morning. I have a problem with this:
Find the maximum and minimum distances from the origin to the curve* $$g\left(x,y\right)=5x^{2}+6xy+5y^{2}$$
I have done this:
Function to optimize:$f\left(x,y\right)=x^{2}+y^{2}$
Restriction: $g\left(x,y\right)=5x^{2}+6xy+5y^{2}=8 $
Applying Lagrange multipliers: $\nabla f\left(x,y\right)=\lambda\nabla g\left(x,y\right)$
Then, $\nabla f\left(x,y\right)=2x\hat{i}+2y\hat{j}$ and $\lambda\nabla g(x,y)=\lambda(2x+\frac{6}{5}y)\hat{i}+\lambda\left(2y+\frac{6}{5}x\right)\hat{j} $
Making the ecuation system:
$\begin{cases} 2x=(2x+\frac{6}{5}y)\lambda\\ 2y=(2y+\frac{6}{5}x)\lambda\\ x^{2}+\frac{6}{5}xy+y^{2}=8 \end{cases}$
But I have serious problem solving the system. Any suggestions?

Have you tried a software like Mathematica? There are four solutions to your system: $x=y=Sqrt[5/2]$ and the negative of that root (both with $λ=5/8$), and $x=-y=Sqrt[10]$ and the negative of that root (both with $λ=5/2$).