I need to describe all functions $G(x,y)$ that have following property:
for some $a,b$ the integral $$ \int_a^b G(x,t) G(x,s) dx $$ is known function $F$ of one variable $s \cdot t$, i.e. \begin{gather} F(s \cdot t)=\int_a^b G(x,t) G(x,s) dx. \qquad (1) \end{gather}
The function $G$ is good function,for example it can be expanded in power series in two variables. The limits $a,b$ and the function $F$ are known.
My attempt. Differentiate $(1)$ by $t$ and $s:$
\begin{gather*} F'(st) s=\int_a^b \frac{\partial G(x,t)}{\partial t} G(x,s) dx,\\ F'(st) t=\int_a^b G(x,t) \frac{\partial G(x,s)}{\partial s} dx. \end{gather*}
Then
\begin{gather*} F'(st) s t =\int_a^b t \frac{\partial G(x,t)}{\partial t} G(x,s) w(x) dx,\\ F'(st) t s=\int_a^b s G(x,t) \frac{\partial G(x,s)}{\partial s} w(x) dx, \end{gather*}
or $$ \int_a^b \left( t \frac{\partial G(x,t)}{\partial t} G(x,s) - s G(x,t) \frac{\partial G(x,s)}{\partial s}\right) dx=0. $$
Can I conclude now that $$ t \frac{\partial G(x,t)}{\partial t} G(x,s) - s G(x,t) \frac{\partial G(x,s)}{\partial s}=0? $$
If not, the how to describe all such function $G(x,y)?$
Suppose $G(x,t)>0$ and $a,b>0$. From $$ t \frac{\partial G(x,t)}{\partial t} G(x,s) - s G(x,t) \frac{\partial G(x,s)}{\partial s}=0 $$ one has $$ \frac{t}{G(x,t)} \frac{\partial G(x,t)}{\partial t} = \frac{s}{G(x,s)} \frac{\partial G(x,s)}{\partial t} $$ which holds for arbitrary $s,t\in[a,b]$. So $$ \frac{t}{G(x,t)} \frac{\partial G(x,t)}{\partial t} \equiv f(x) $$ is independent of $t$. This gives $$ \frac{\partial \ln G(x,t)}{\partial t}=\frac{f(x)}{t}. $$ So $$ \ln G(x,t)-\ln G(x,a)=f(x)(\ln t-\ln a)$$ or $$ G(x,t)=G(x,a)\left(\frac{t}{a}\right)^{f(x)}. $$