Consider $ E[u]= \int^1_0 \big(u'(x)\big)^2+\big(u(x)\big)^2-2f(x)u(x) dx.$
Calculate the variational derivation for a function $v$; in other words, calculate $\frac{d}{d\epsilon}E[u+\epsilon v]$ at $\epsilon=0$.
*$\frac{d}{d\epsilon}$ is the derivative with respect to $\epsilon$.
My work:
$\frac{d}{d\epsilon}E[u+\epsilon x] = \frac{d}{d\epsilon}\int^1_0(u+\epsilon v)'(x)^2+(u+\epsilon v)(x)^2-2f(x)(u+\epsilon v)(x) dx$ . I moved the derivative sign inside the integral and split the integral up, like so: $\int^1_0\frac{d}{d\epsilon}(u+\epsilon v)'(x)^2 dx+\int^1_0\frac{d}{d\epsilon}(u+\epsilon v)(x)^2-2\int^1_0\frac{d}{d\epsilon}f(x)(u+\epsilon v)(x) dx$. Here is where I get stuck; I'm not sure how to take the derivatives and simplify.
It may be very unclear what stands for $$ \frac{d}{d\epsilon}(u+\varepsilon v)'(x)^2 $$ Let's put the $x$ argument along with the derivative inside. Since the derivative is taken with respect to $x$, $\epsilon$ stands as a constant term. $$ \frac{d}{d\epsilon}\big(u'(x)+\varepsilon v'(x)\big)^2. $$ Now it is clear that $$ \frac{d}{d\epsilon}(u'(x)+\varepsilon v'(x))^2 = \frac{d}{d\epsilon}\Big(\big(u'(x)\big)^2+2\varepsilon u'(x)v'(x) + \epsilon^2\big(v'(x)\big)^2\Big) = 2 u'(x) v'(x) + 2 \epsilon \big(v'(x)\big)^2. $$ When $\epsilon = 0$ the last term vanishes. So $$ \left.\frac{d}{d\epsilon} \int_0^1 (u'(x) + \epsilon v'(x))^2 dx\right|_{\epsilon=0} = \int_0^1 2 u'(x) v'(x) dx $$ This integral is often taken by parts to eliminate $v'(x)$: $$ \int_0^1 2 u'(x) v'(x) dx = 2u'(x)v(x)\Big|_0^1 - 2\int_0^1 u''(x)v(x)dx. $$ I hope that you can finish the exercise by yourself now. The final answer should have the form of $$ \frac{d}{d\epsilon}E[u+\epsilon v] = 2u'(x)v(x)\Big|_0^1 + \int_0^1 [\bullet] v(x) dx $$
Addition. The variational derivative is $$ \frac{d}{d\epsilon}E[u+\epsilon v] = 2u'(x)v(x)\Big|_0^1 + \int_0^1 [-2u''(x)+2u(x)-2f(x)] v(x) dx. $$ Now we want to solve $$ \frac{d}{d\epsilon}E[u+\epsilon v]\big|_{\epsilon = 0} = 0 $$ for every test function $v(x)$. Let's start with $v(x) = \delta(x - a), 0 < a < 1$. The $2u'(x)v(x)\Big|_0^1$ term vanishes since $v(0) = v(1) = 0$. $$ \int_0^1 [-2u''(x)+2u(x)-2f(x)] \delta(x-a) dx = -2u''(a)+2u(a)-2f(a) = 0 $$ It means that for every inner point of $[0, 1]$ the following equation should hold $$ u''(x) - u(x) = f(x),\qquad 0 < x < 1. $$ If $u(x)$ satisfies this equation then the last integral in the variational derivatives is zero, i.e. for arbitrary $v(x)$ $$ \frac{d}{d\epsilon}E[u+\epsilon v]\big|_{\epsilon = 0} = 2u'(x)v(x)\Big|_0^1 = 2u'(1) v(1) - 2u'(0)v(0). $$ By choosing $v(x) = x$ and $v(x) = 1 - x$ both of the $u'(0)$ and $u'(1)$ should be zero.
Result. $$ \frac{d}{d\epsilon}E[u+\epsilon v]\big|_{\epsilon = 0} = 0 $$ for every $v(x)$ is equivalent to following problem for $u(x)$: $$ u''(x) - u(x) = f(x),\qquad 0 < x < 1\\ u'(0) = u'(1) = 0 $$