I have the following functional \begin{align*} F[f]=\int f(x) \log(g(x)) dx \end{align*} where $g(x)$ is given by convolution $g(x)=y(x) * f(x)=\int y(\tau) f(x-\tau) d\tau$, so \begin{align*} F[f]&=\int f(x) \log(g(x)) dx=F[f]=\int f(x) \log(y(x) * f(x)) dx\\ &=\int f(x) \log \left(\int y(\tau) f(x-\tau) d\tau \right) dx \end{align*} assume that $y(x)$ is fixed.
My Goal is to find functional derivative or Gateaux derivative with respect to $f$.
My issue is that I don't know how to deal with convolution when I take the derivative.
What I did
The variation is given by $\left[ \frac{d}{d\epsilon} F[f+\epsilon \theta] \right]_{\epsilon=0} $.
My question is do I work with the convolution term or no?
Here are the two possibilities
1) Don't work with the convolution term \begin{align*} \left[ \frac{d}{d\epsilon} F[f+\epsilon \theta(x)] \right]_{\epsilon=0} =\left[ \frac{d}{d\epsilon} \int (f(x)+\epsilon \theta(x)) \log(g(x)) dx \right]_{\epsilon=0}= \int \theta \log(g(x)) dx \end{align*}
2) Work with the convolution term
\begin{align*} \left[ \frac{d}{d\epsilon} F[f+\epsilon \theta] \right]_{\epsilon=0} =\left[ \frac{d}{d\epsilon} \int (f(x)+\epsilon \theta(x)) \log \left(\int y(\tau) (f(x-\tau)+\epsilon \theta(\tau)) d\tau \right) dx \right]_{\epsilon=0} \end{align*}
In the integral inside convolution should it be $\theta(\tau)$ of $\theta(x)$??
Which way is correct? if the second way is correct how do I proceed next?
I was also thinking about using chain rule of derivatives for functional \begin{align*} \frac{\delta F[G[f]]}{\delta f}=\frac{\delta F[G[f]]}{\delta G}\frac{\delta G[f]}{\delta f} \end{align*} but I don't think convolution is a functional, so I am not sure if this applies. Thank you fro any help in advance.
First of all: What is $y$? I assume, just some suitable fixed function?
Your approach 1) is certainly wrong since it ignores the dependence of $g$ on $f$.
Now your second approach is almost correct. However neither $\theta(x)$ nor $\theta(\tau)$ is correct. Instead you have to take $\theta(x-\tau)$, i.e. exactly the same argument that you insert into $f$. Now you can split the integral into to terms: $$\frac{d}{d\varepsilon}\int f(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx\quad\\\quad+\frac d{d\varepsilon}\int\varepsilon\theta(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx$$
Now we first treat the first term which is equal to $$\int\frac{f(x)\cdot(y*\theta)(x)}{\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau}dx$$ and setting $\varepsilon=0$ yields $$\int\frac{f(x)\cdot(y*\theta)(x)}{(y*f)(x)}dx=\int\frac{f(x)\cdot(y*\theta)(x)}{g(x)}dx.$$
Now the second term is equal to $$\int\theta(x)\log\left(\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau\right)dx+\int\frac{\varepsilon\theta(x)\cdot(y*\theta)(x)}{\int y(\tau)(f(x-\tau)+\varepsilon\theta(x-\tau))\,d\tau}dx$$ and setting $\varepsilon=0$ yields $$\int\theta(x)\log(g(x))\,dx.$$
Hence the final result should be $$\int\left(\frac{f(x)\cdot(y*\theta)(x)}{g(x)}+\theta(x)\log(g(x))\right)dx.$$
One additional remark: I have just seen that you write $g(x)=f(x)*y(x)$. You should avoid that and write $g(x)=(f*y)(x)$ instead.
To answer your question in the comment: yes you can express it in this form. This works as follows. First, we only consider the first term of the final result. $$\int\frac{f(x)\cdot(y*\theta)(x)}{g(x)}dx=\int\frac{f(x)}{g(x)}\int y(\tau-x)\theta(\tau)\,d\tau\,dx.$$ Now we change the order of integration (of course you should justify this step rigorously but that needs more information about $y$ and the domain of $F$). Thus we obtain $$\int\theta(\tau)\int \frac{f(x)}{g(x)}y(\tau-x)\,dx\,d\tau.$$ Of course we can relabel the integration variables and thus, by combining it with the second term of the result (which is already in the desired form) we obtain $$\int\left(\int \frac{f(\tau)}{g(\tau)}y(x-\tau)\,d\tau+\log(g(x))\right)\theta(x)\,dx$$
Perhaps there is some neat way to rewrite the inner integral but I fail to see it.