Let $f(x,y) = x^2 +y^2$ with $ g(x,y) = 6x^2+4xy+9y^2-50$.
$\Longrightarrow(2x,2y) + \lambda \cdot (12x+4y,18y+4x)$
So I have to solve $$2x + 12x \lambda +4y \lambda = 0,$$
$$2y+4x\lambda+18y\lambda=0.$$
I am not really sure on how to proceed here. Do I try different values for $x$ or $y$ which are acceptable under the restriction $g(x,y)$ ?
I tried to multiply the first eq. with $y$ and the second with $x$ but didn't get far in solving that...
I presume that you want to find the extrema for $f(x,y) = x^2 +y^2$ given the constraint $6x^2+4xy+9y^2=50$.
So let $g(x,y)=6x^2+4xy+9y^2-50$ (as you have proposed).
Then we need to solve for $\nabla f = \lambda \nabla g$. This means $$ \langle 2x, 2y\rangle = \lambda \langle 12x + 4y, 4x+18y \rangle, $$ so we have $$ \begin{align*} 2x &=\lambda (12x+4y), \\ 2y &= \lambda (4x+18y). \\ \end{align*} $$ If $\lambda =0$, then $x=y=0$ by the two previous equations. So substituting $x=y=0$ into the constraint equation, we would then have $0=50$, which is a contradiction (this is not possible). So $\lambda\not=0$.
Expand the two equations to obtain $$ \begin{align*} 2x &=12x \lambda+4y\lambda, \\ 2y &= 4x\lambda + 18y\lambda. \\ \end{align*} $$ So $$ 2x-12x \lambda = 2x(1-6\lambda)=4y\lambda. $$ Since $\lambda\not=0$, $$ \boxed{y=\dfrac{x(1-6\lambda)}{2\lambda}}. $$ Similarly from the second equation, we obtain that $2y(1-9\lambda)=4x\lambda$. Substitute the above $y$ into $2y(1-9\lambda)=4x\lambda$ to obtain $$ \frac{x(1-6\lambda)}{\lambda}(1-9\lambda)=4x\lambda. $$ Clear the denominator to get $x(1-6\lambda)(1-9\lambda)=4x\lambda^2$. Move the term on the right to the left-hand side to obtain $x(1-15\lambda+54\lambda^2-4\lambda^2)=0$, or $$ x(5\lambda-1)(10\lambda-1)=0. $$ So $$ x=0 \hspace{4mm}\mbox{ or }\hspace{4mm} \lambda =\frac{1}{5} \hspace{4mm}\mbox{ or }\hspace{4mm} \lambda = \frac{1}{10}. $$ If $x=0$, then using the boxed equation, we see that $y=0$. This is a contradiction as we have seen above.
If $\lambda =1/5$, then substitute this into the boxed equation to get $$ y=\dfrac{x\left(1-\frac{6}{5}\right)}{\frac{2}{5}} = -\frac{x}{2}. $$ Substitute $y=-x/2$ into the constraint equation to obtain $$ 6x^2-2x^2+\frac{9x^2}{4}=50. $$ This simplifies as $25x^2=200$, or $x^2=8$. So $x=\pm 2\sqrt{2}$. $$ \begin{align*} \mbox{ If } x &= 2\sqrt{2}, \mbox{ then } y=-\sqrt{2}. \\ \mbox{ If } x &= -2\sqrt{2}, \mbox{ then } y=\sqrt{2}. \\ \end{align*} $$ So we obtain two critical points of interest: $$ \boxed{ (2\sqrt{2},-\sqrt{2})} \mbox{ and } \boxed{(-2\sqrt{2},\sqrt{2})}. $$
If $\lambda=1/10$, then substitute this into the boxed equation to obtain $$ y=\dfrac{x\left( 1-\frac{6}{10}\right)}{\frac{2}{10}} = 2x. $$ Substitute $y=2x$ into the constraint equation to obtain $$ 6x^2+8x^2+9(4x^2)=50, $$ or $x^2=1$, i.e., $x=\pm 1$. $$ \begin{align*} \mbox{ If } x &= 1, \mbox{ then } y=2. \\ \mbox{ If } x &= -1, \mbox{ then } y=-2. \\ \end{align*} $$ So we obtain two more points of interest: $$ \boxed{ (1,2)} \mbox{ and } \boxed{(-1,-2)}. $$ Finally, substitute all four points into $f(x,y)$ to obtain $$ \begin{align*} f(2\sqrt{2},-\sqrt{2}) &= 8+2 =10, \\ f(-2\sqrt{2},\sqrt{2}) &= 8+2 =10, \\ f(1,2) &= 1+4=5, \\ f(-1,-2) &= 1+4 = 5. \\ \end{align*} $$ We conclude that $\color{blue}{(1,2)}$ and $\color{blue}{(-1,-2)}$ give us $\color{blue}{\textbf{absolute minimum}}$ while $\color{green}{(2\sqrt{2},-\sqrt{2})}$ and $\color{green}{(-2\sqrt{2},\sqrt{2})}$ give us $\color{green}{\textbf{absolute maximum}}$ for $f$ with respect to the constraint $6x^2+4xy+9y^2=50$.