I was playing around with another example that I made up where I am trying to maximize the area of a triangle inscribed in a circle of radius. I want to do the problem using the method of Lagrange Multipliers.
Attempt: Consider the circle of radius 2 given by $x^2+y^2=4$. The function we are trying to maximize is $A(x,y) = \frac{1}{2}xy$. Let $h(x,y)=4-x^2-y^2$.
Then, $\nabla h(x,y) = [-2x \ \ -2y ]$ and $\nabla A(x,y) = [\frac{y}{2} \ \ \frac{x}{2}]$. We want, $ [\frac{y}{2} \ \ \frac{x}{2}]= \lambda[-2x \ \ -2y ] $. So, $[x \ \ y]=[-4\lambda y \ \ -4\lambda x]$.
Hence, $(-4\lambda y)^2+(-4\lambda x)^2 = 4 \Rightarrow 4\lambda^2(y^2+x^2)=1 \Rightarrow x^2+y^2 = \frac{1}{4\lambda^2} \Rightarrow \lambda = \frac{1}{4}$. It then follows that $[x \ \ y] = [-0.5y \ \ -0.5x]$. And, $h(-0.5y,y) = 4-\frac{5}{4}y^2 = 0 \Rightarrow [x \ \ y] = [\frac{-4}{\sqrt{5}} \ \ \frac{4}{\sqrt{5}}]$. But then $A(x,y)<0$.
Question: What am I doing wrong?
You found the minimum area (if you accept negative area), which is one of the extreme solutions. What about $\lambda = -\frac{1}{4}$, though? You concluded from $\frac{1}{4\lambda^2}=4$ that $\lambda = \frac{1}{2}$ (I think you meant $\lambda = \frac{1}{4}$), but there are two roots.
Also, are you sure that you are solving the problem you want to be solving? As it is, you're maximizing the area of a right triangle with one vertex on the origin. The question asks for the largest triangle in a circle, so either you haven't described the problem in enough detail, or you're solving the wrong problem.
Finally, an alternative approach: since your point $(x,y)$ is on a circle of radius $2$, you could let $(x,y) = (\sqrt2\cos\theta,\sqrt2\sin\theta)$. Then the area is $$A=\frac{1}{2}xy=\frac{1}{2}(\sqrt2\cos\theta)(\sqrt2\sin\theta)=\sin\theta\cos\theta$$ Now use $\sin 2\theta = 2\sin\theta\cos\theta$, and the answer follows.