The question goes something like this:
A simple closed curve in the $x$-$y$ plane is specified in terms of $\theta$ which ranges from $0$ to $2\pi$. The area enclosed by the curve is $$ A[x,y] = \frac12\int_0^{2\pi} (xy' - yx') \mathrm d \theta. $$ Use this expression and the method of Lagrange multipliers to determine the curve that maximizes the enclosed area for fixed length.
Noting that the length is fixed, we have $$ l = L[x,y] = \int_0^{2\pi} \sqrt{x'^2 + y'^2} \mathrm d\theta. $$ I then considered the functional $$ h[x,y;\lambda) = (xy' - yx') - \lambda\sqrt{x'^2 + y'^2}, $$ which must satisfy the Euler-Lagrange equations $$ \frac{\partial h}{\partial x} - \frac{\mathrm d}{\mathrm d\theta}\left( \frac{\partial h}{\partial x'}\right) = 0, \quad \frac{\partial h}{\partial y} - \frac{\mathrm d}{\mathrm d\theta}\left( \frac{\partial h}{\partial y'}\right) = 0. $$ I tried the standard method of noting that $h$ has no explicit $\theta$ dependence to get $$ h - x' \frac{\partial h}{\partial x'} - y' \frac{\partial h}{\partial y'} = \text{constant}, $$ but the terms cancelled out on the LHS and I got $0=0$, which is not helpful. Trying to directly solve the E-L equations themselves seems even more daunting. Is there something I'm missing here?
I am aware that the enclosed area problem can be solved with just a single dependent variable $y(x)$, but this is not what the question is asking for.
Solving the E-L equations is not that difficult: \begin{align} \frac{\partial h}{\partial x} - \frac{d}{d\theta}\left( \frac{\partial h}{\partial x'}\right) = 0 &\implies y'- \frac{d}{d\theta}\left(-y-\frac{\lambda x'}{\sqrt{x'^2+y'^2}}\right) =0 \\ &\implies \frac{d}{d\theta}\left(2y+\frac{\lambda x'}{ \sqrt{x'^2+y'^2}}\right) =0 \\ &\implies 2y+\frac{\lambda x'}{ \sqrt{x'^2+y'^2}}=C_1. \tag{1} \end{align} A similar calculation shows that $$ \frac{\partial h}{\partial y} - \frac{d}{d\theta}\left( \frac{\partial h}{\partial y'}\right) = 0 \implies -2x+\frac{\lambda y'}{ \sqrt{x'^2+y'^2}}=C_2. \tag{2} $$ Multiplying $(1)$ by $y'$ and $(2)$ by $x'$, and subtracting the latter from the former, we obtain $$ 2yy'+2xx'=C_1y'-C_2x' \implies y^2-C_1y+x^2+C_2x=C_3. \tag{3} $$ Completing the squares and renaming the constants, we finally obtain the well known result that the curve that maximizes the area enclosed by it is a circle: $$ (x-a)^2+(y-b)^2=r^2. \tag{4} $$