I had the following solution for this problem. I hope I'm not missing something important.
Find extremals for the functional $J[y]=\displaystyle\int_0^1(xy+y^2-2y^2y')dx$ with $y(0)=1$ and $y(1)=2$. For the Euler-Lagrange equation we have $$0=F_y-\dfrac{d}{dx}(F_y')=x+2y-4yy'-(-4yy')=x+2y\Rightarrow y=-\dfrac{x}{2}$$ which is inconsistent with the boundary conditions so there is no extremal for $J$.
Yes, that's right. In particular, you're implicitly performing the extremization over some function space for which $J[y]$ is well-defined, e.g. $H^1([0,1])$. Suppose for contradiction that you have an extremum $y(x)$ that satisfies your boundary conditions. Then for any $\delta y(x) \in H^1([0,1])$ with $\delta y(0) = \delta y(1) = 0$, you've computed that $$\lim_{t\to 0} \frac{d}{dt} J[y + t\delta y] = \int_0^1 (x+2y)\delta y,$$ and since $y$ is continuous and not equal to $-x/2$ everywhere, you can construct a $\delta y$ (using a bump function e.g.) which contradicts the extremality of $y$.
If you imagine actually performing extremization (for instance, by repeatedly perturbing $y$ in a direction $\delta y$ that decreases $J$) you'll see that $y$ "looks more and more" like $y(x)=-x/2$, with derivatives that become larger and larger near the boundaries. As Ninad Munshi points out, since $H^1([0,1])$ is not compact, this process won't converge to a minimum in $H^1([0,1])$ (it should be easy to visualize that the sequence is "converging" to $y(x) = -x/2$ with two discontinuities at the endpoints).