I am doing some work on Kernel based machine learning where I encountered the following functional - $$\frac{1}{m}\sum_{i=0}^m(y_i-f(x_i))^2+\gamma\lVert f\rVert^2$$ Here $\gamma$ is a positive real number, $x_i,y_i$ are given (these are the training data), and norm is of the function space(Reproducing Kernel Hilbert Space). Now I need to find a function that minimizes the given functional. This is what I came across - "To minimize the given functional, we take the functional derivative with respect to $f$, apply it to an element $\bar{f}$ of the function space, and set it equal to $0$. We obtain $$\frac{1}{m}\sum_{i=0}^m(y_i-f(x_i))^2\bar{f}(x_i)-\gamma\left<f,\bar{f}\right> = 0$$ How do we arrive at this equation, or how do we take functional derivative?
Reference - "The Mathematics of Learning:Dealing with Data, Tomaso Poggio and Steve Smale"
The term "functional derivative" is, in my opinion, quite old-fashioned and it can be misleading. In this context, what is meant is a directional derivative in the direction of $\overline f$. (This is also known as "Gateaux derivative"). The recipe is: replace any occurrence of $f$ with $f+\epsilon \overline f$ and then take a derivative at $\epsilon =0$. For the present question, $$\begin{split} &\left.\frac{\partial}{\partial \epsilon}\right|_{\epsilon =0}\left( \frac1m \sum_i (y_i - f(x_i) - \epsilon \overline f(x_i))^2 +\gamma\| f+\epsilon \overline f\|^2\right)\\& = -\frac2m\sum_i (y_i - f(x_i))\overline f (x_i) + 2\gamma \langle f|\overline f\rangle. \end{split}$$ I have used the formula $\|f\|^2= \langle f | f \rangle $ to compute the derivative of $\| f+\epsilon \overline f\|^2$.