How to calculate this kind of integrals?
$$\int_a^b \left(\int_a^b \frac{f(t) \overline{f(s)}}{1-ts} \, ds\right) dt$$
$a=0$, $0<b<1$, $t,s \in [a,b]$ are real, and $f$ "lives" in $C([a,b], \mathbb{C})$
I have to find that it's equal to $\sum_{n=0}^{+\infty} \left|\int_a^b f(t) t^n \, dt\right|^2.$
I just know that $\sum\limits_n (st)^n = \dfrac{1}{1-ts} \dots$
Could someone help me?
You're almost there. You have for $|a|<1$ and $|b|<1$
$$\int_a^b\int_a^b \frac{f(t)\overline{f(s)}}{1-ts}\,ds\,dt= \int_a^b\int_a^b \sum_{n=0}^\infty f(t)t^n\overline{f(s)}s^n\,ds\,dt$$
Next, note that
$$\begin{align} \lim_{N\to\infty}\int_a^b\int_a^b f(t)\overline{f(s)}\,\left(\frac{1-(ts)^{N+1}}{1-ts}\right)\,ds\,dt&=\sum_{n=0}^\infty \int_a^b\int_a^b f(t)t^n\overline{f(s)}s^n\,ds\,dt\\\\ &=\sum_{n=0}^\infty \left(\int_a^b f(t)t^n\,dt\right)\left(\overline{\int_a^b f(t)t^n\,dt}\right)\\\\ &=\sum_{n=0}^\infty \left|\int_a^b f(t)t^n\,dt\right|^2 \end{align}$$
Since $f$ is continuous, then its magnitude is bounded and the Dominated Convergence Theorem guarantees that we can pass the limit under the integral to arrive at
$$\int_a^b\int_a^b \frac{f(t)\overline{f(s)}}{1-ts}\,ds\,dt=\sum_{n=0}^\infty \left|\int_a^b f(t)t^n\,dt\right|^2$$
as was to be shown!