Let $\overline{A}$ be conjugation, $A^T$ be transposition and $A^H$ be conjugation and transposition. Given the usual $L^2(\mathbb{C})$ inner product:
$$\langle f, g \rangle = \int_{-\infty}f(x)\overline{g(x)}\,dx$$
and the energy ($f\in L^2(\mathbb{R}), A,B : L^2(\mathbb{C}) \rightarrow L^2(\mathbb{C})$, $A,B$ linear):
$$E[f] = \langle \mathcal{A}f, \mathcal{B}f\rangle,$$
If $f$ is real I take the functional derivative using the definition:
$$\left\langle\frac{dE}{df},\phi\right\rangle = \frac{d}{d\lambda}E[f+\lambda\phi]|_{\lambda=0}=\frac{d}{d\lambda}\langle A(f+\lambda\phi), B(f+\lambda \phi)\rangle|_{\lambda=0} = \\ \langle Af, B\phi\rangle + \langle A\phi, Bf\rangle = \langle B^HAf, \phi\rangle + \langle \overline{Bf}, \overline{A\phi}\rangle = \langle B^HAf, \phi\rangle + \langle A^T\overline{Bf}, \overline{\phi}\rangle= \langle (B^HA+A^T\overline{B})f, \phi\rangle,$$
then $\frac{dE}{df} = (B^HA+A^T\overline{B})f$. First of all, is this derivative correct? I think I have some mistake since I expected to get something symmetric from it, e.g. $(B^HA+A^HB)f$. Also how would the functional derivative for a complex $f$ be defined. Would I stop at the point:
$$\left\langle\frac{dE}{df},\phi\right\rangle = \langle B^HAf, \phi\rangle + \langle A^T\overline{Bf}, \overline{\phi}\rangle,$$
and then term $\frac{dE}{df}=B^HAf$ and $\frac{dE}{d\overline{f}} = A^T\overline{Bf}$? Or would I split up the imaginary and real terms and then have $\phi_R, \phi_I$ for those and take derivatives with respect to those? References are more than welcome.