What are some general properties of $F(x, z) = \int f(x, y) f(y, z) dy$ ?
For example, what can be said about the relation between $F(x,z)$ and $f(x,z)$?
Optical theorem in physics of wave scattering has this form. What other theorems in natural sciences and mathematics also have this form?
On
It is hard to give a detailed answer without knowing which kind of properties you're seeking. One idea: it may be helpful to think of the analogous definition for $x$, $y$, and $z$ discrete, taking values in $\{1,\ldots, n\}$. In this case $f$ is a matrix and $F$ is just $f^2$, the matrix product of $f$ with itself. Depending on your situation this may yield some insights. For example, if $f$ is a symmetric matrix then $f^2$ is positive semidefinite, and a similar statement should hold for the case of continuous variables if the terms are reinterpreted appropriately.
I will instead consider the operation taking $f$ and $g$ to $F(x,z)=\int f(x,y) g(y,z)d y$. Everything said below is under mild assumptions, measurability or continuity.
The operation is associative.
As Noah observed, when the integral is a sum, it is matrix product.
For symmetric functions: it induces inner product on space of continuous symmetric bivariate real functions.
The formula reminds me of Möbius transform and incidence algebra, I think it is a continuous form of this.
It reminds me also of composition of relations and composition of profunctors in category theory
Also faintly reminds me of convolution