here it is
Find all the possible extremals:
$$J[u]=\int_0^1 \int_0^{1} (\frac{\partial u^2(x,y)}{\partial x}+\frac{\partial u^2(x,y)}{\partial y}) dxdy$$
subject to constraint: $$I[u]=\int_0^1 \int_0^{1} u^{2}(x,y) dxdy-1=0$$ and $$A=\left \{u\epsilon C^2 | u(x,0)=u(x,1)=0, x\epsilon [0,1]\right \} $$
*I tried solving it by setting $$R[u]=J[u]+\lambda I[u]$$ and then after some calculations I ended up here $$ \nabla^{2} u(x,y)= \lambda u(x,y) $$ which I tried to solve it with seperation of variables and I failed, pls help.
So after correction from the link you have $J[u]=\int_{[0,1]^2} |\nabla u|^2 dx dy$. You can compute:
$$J[u+v]=\| \nabla u \|_{L^2}^2 + 2 \langle \nabla u,\nabla v \rangle + \| \nabla v \|_{L^2}^2$$
where the inner product is the $L^2$ inner product.
Therefore $J'(u)[v] = 2 \langle \nabla u,\nabla v \rangle$. Similarly $I'(u)[v]=2 \langle u,v \rangle$. Now the first order necessary condition amounts to finding $u^*$ such that the kernel of $I'(u^*)$ (in the domain $A$) is contained in the kernel of $J'(u^*)$ (again in the domain $A$). It will probably help to integrate $\langle \nabla u,\nabla v \rangle$ by parts to get a simpler expression for this.