How can I calculate the functional derivative of this functional?
$$F[x](t) = \int_{0}^{t}x(t_1)a(t_1)\left \{ \int_{0}^{t_1}x(t_2)b(t_2) \,dt_2\right \} dt_1 .$$
Where $a(t)$ and $b(t)$ are real fixed functions. Following the usual method, I'm getting
$$\frac{\delta F\left [ x \right ]}{\delta x(s) } = \int_{0}^{s}\left [ a(t')b(s)+a(s)b(t') \right ]x(t')dt'.$$
which seems odd since that is also a functional, and I think that I should be getting a function.
Use the identity \begin{align} \frac{\delta x(t)}{\delta x(s)} = \delta(t-s) \end{align} to obtain \begin{align} \frac{\delta}{\delta x(s)}\Big(x(t_1)x(t_2)a(t_1)b(t_2)\Big) &= \delta(t_1-s)x(t_2)a(t_1)b(t_2) + x(t_1)\delta(t_2-s)a(t_1)b(t_2) \end{align} from which it follows that \begin{align} \frac{\delta F}{\delta x(s)}[x] = a(s)\int_0^s x(t_2)b(t_2)\,dt_2 + b(s)\int_s^tx(t_1)a(t_1)\, dt_1 \end{align} where we have assumed $0<s<t$ and $0<s<t_1$.