Let $x$ and $y$ be real numbers. Find the maximum value of $2x^2 - 3xy - 2y^2$ subject to the condition that $25x^2 - 20xy + 40y^2 = 36$.
So first I factored out $2x^2 - 3xy - 2y^2 = (x-2y)(2x+y)$, and because "maximum" kinda reminded me of the AM-GM, I went ahead to use the formula and got $(\frac{3x-y}{2})^2 \ge 2x^2 - 3xy - 2y^2.$ But that didn't seem to be helping me much so I moved $36$ to the LHS of the equation given so that $25x^2 - 20xy + 40y^2 -36 = 0$, which can also be written as $25x^2 - 20xy + 4(10y^2-9) = 0$ or $25x^2 - 20xy + 4(\sqrt{10}y + 3)(\sqrt{10}y - 3)=0$. The latter isn't quite nice because it's almost there but not quite — I don't think I can factor it unless $20xy$ were $20\sqrt{10}xy$, which isn't the case.
So at this point, I'd really like some help :)
For a calculus-free solution, start with the factorization $2x^2 - 3xy - 2y^2 = (x-2y)(2x+y)$ and make the substitution $u = x-2y$, $v = 2x+y$. Then the condition $$25x^2 - 20xy + 40y^2 = 36$$ turns into $$9u^2 + 4v^2 = 36$$ so we are maximizing $uv$ subject to this condition.
Now we can apply AM-GM: $9u^2 + 4v^2 \ge 2 \sqrt{(9u^2)(4v^2)} = 12|uv|$, or to rearrange this a bit, $$uv \le \frac{9u^2+4v^2}{12} = \frac{36}{12} = 3.$$ This upper bound is achieved when $9u^2 = 4v^2$, the equality condition of AM-GM (and when $uv = |uv|$, because $uv \le |uv|$ was the other inequality we used). If we wanted to, we could solve this to find the values of $x$ and $y$ that give us the maximum value of $3$.