Consider the following problem:
\begin{eqnarray} &\inf_{f} {\int\limits_0^1 g(x)f(x) dx}\\ &\textrm{s.t.} \quad \int\limits_0^{2} g(x)(f\ast f(x)) dx=c\\ & \quad f:[0,1] \rightarrow [0,1] \end{eqnarray} where $f\ast f(x)$ is the convolution of $f$ with itself, $c>0$ is low enough so that the feasible set is non-empty.
I plan to start working with the first variation, but wanted to pause and check here if there is a standard method of dealing with this kind of problem (maybe convolution suggests a move to the frequency domain?). Thank you.
Note: Original problem had $g(x)=xe^{-x}$, but the solution is more general.
The first variation needs two perturbation $\eta_1$ and $\eta_2$ and they are dependent on each other. In effect this gives the Lagrange multiplier.
Define $g(x):=xe^{-x}$. In fact, it will be clear from the derivation below that this function is a red herring so long as it is not zero. We have either $$\label{conVar}\int g(x)(f*\eta)(x)\,dx=0 \tag1$$ or the stationary function $f$ satisfies $$\label{var}\int g(x)(\eta(x)+\lambda f*\eta(x))\,dx=0,\tag2$$ for all admissible $\eta$, where $\lambda$ is some constant, i.e., the Lagrange multiplier. Denote the Fourier transform of $f$ by $\mathcal F[f]$ and the complex conjugation of $g$ by $\bar g$. By the Plancherel theorem, we have for Equation \eqref{conVar} \begin{align} 0&=\int g(x)(f*\eta)(x))\,dx \\ &=\int \overline{\mathcal F[g]}(k)\,\mathcal F[f*\eta](k)\,dk \\ &=\int \overline{\mathcal F[g]}(k)\,\mathcal F[f](k)\mathcal F[\eta](k)\,dk \end{align} for all admissible $\eta$. So $$\mathcal F[f]=0 \Longleftrightarrow f=0.$$ Depending on whether $c=0$ or not, $f=0$ is a stationary function or not.
By the same token, Equation \eqref{var} gives \begin{align} 0&=\int g(\eta(x)+\lambda f*\eta(x))\,dx \\ &=\int g\big((\delta+\lambda f)*\eta(x)\big)\,dx \\ &=\int \overline{\mathcal F[g]}(k)\,\mathcal F[(\delta+\lambda f)*\eta(x)](k)\,dk \\ &= \int \overline{\mathcal F[g]}(k)\,\mathcal F[(\delta+\lambda f)](k) \,\mathcal F[\eta(x)](k)\,dk \\ &= \int \overline{\mathcal F[g]}(k)\,(1+\lambda\mathcal F[f])(k) \,\mathcal F[\eta](k)\,dk \end{align} for every admissible $\eta$. Since $\eta$ is arbitrary, $$1+\lambda\mathcal F[f]=0 \Longleftrightarrow f(x)=-\frac1\lambda\delta(x).$$ $\therefore f:[0,1] \not\rightarrow [0,1].$