Use the method of Lagrange multipliers to find the function $y(x)$ that makes the functional $S[y]=\int_{1}^{2}x^2y^2dx$ stationary subject to the two constraints $\int_{1}^{2}xydx=1$ and $\int_{1}^{2}x^2ydx=2$. Calculate the stationary value of $S[y]$.
Here's my work:
Consider the functional $S[y]=\int_{1}^{2}x^2y^2dx$ stationary subject to the two constraints $\int_{1}^{2}xydx=1$ and $\int_{1}^{2}x^2ydx=2$.
Then $\overline{S}[y]=\int_{1}^{2}(x^2y^2-\lambda y)dx, y(1)=y(2)=0$, where $\lambda$ is the Lagrange multiplier.
By definition, the Euler-Lagrange equation is $\frac{d}{dx}(\frac{\partial F}{\partial y'})-\frac{\partial F}{\partial y}=0, y(a)=A, y(b)=B$ for the functional $S[y]=\int_{a}^{b}F(x, y, y')dx, y(a)=A, y(b)=B$.
Note that $F(x, y, y')=x^2y^2-\lambda y$.
This gives $\frac{\partial F}{\partial y'}=0, \frac{\partial F}{\partial y}=2x^2y-\lambda$ and $\frac{d}{dx}(\frac{\partial F}{\partial y'})=0$.
Thus, the Euler-Lagrange equation is $\frac{d}{dx}(\frac{\partial F}{\partial y'})-\frac{\partial F}{\partial y}=0\implies -2x^2y+\lambda=0\implies x^2y=\frac{\lambda}{2}$.
This has the general solution $y(x)=\frac{\lambda}{2x^2}$.
From here I'm stuck. I don't know if the work above is correct or not up to here, but from here, how should I find the function $y(x)$ by using the method of Lagrange multipliers and how to calculate the stationary value of $S[y]$?
Something that simplifies this problem substantially is that unlike most variational problems this does not involve derivatives $y'$ in either the primary functional or the constraints, which means that it doesn't yield and ODE.
In this case you need a Lagrange multiplier for each, whereas you seem to have only included a single constraint. So the equation is $$ F_y+\lambda_1 S_y + \lambda_2 T_y = 0$$ where $S$ and $T$ are the constraints. By the way, I'm trying to match the notation of Gelfand and Fomin.
In this case $F_y=2x^2y$, $S_y=x$ and $T_y=x^2$. So we now have
$$ 2x^2y + \lambda_1 x + \lambda_2 x^2=0$$
You can now solve for $y$ and based on your two conditions figure out what the values of $\lambda_1$ and $\lambda_2$ have to be.