Let $g(x)$ be a non-negative real function over $[-T,T]$, find the (may be complex) function $f(x)$ that maximizes $F(f)$: $$F(f)=\frac{\left|\int_{-T}^{T}g(x)f(x) \, dx\right|^2}{\int_{-T}^T \left|f(x)\right|^2 \, dx + k\int_{-T}^T g(x)\left|f(x)\right|^2 \, dx}$$
If the second term in the denominator does not exist, it is easy to find the optimum $f$ by using Cauchy-Schwarz inequality. However, I have no idea what should I do when it exists. Any idea?
Assuming that all terms exist, take the logarithm (and separate all terms) and then the functional derivative and set to zero. It can be done since the logarithm is strictly monotonic so it preserves maxima and minima.