I found a similar system of integral equations in a paper. It says that it can be solved by differentiating and then using standard techniques. My question is, how can I differentiate such a system in general? Do I have to differentiate for $x$ or $y$ or for both?
$$f(y)+f(x) + xh(x) + \int_x^y zg(z)dz=a, \qquad \forall x <y $$
$$f(x)+h(x) + \int_x^\infty g(z)dz=1 $$
$$f(x), h(x),\qquad g(z) \geq 0$$
where $x$ and $y$ are variables in $\mathbb{R}$ and $f,g,h$ are functions to be determined.
I would be very glad for any help on how to proceed here!
You have three unknowns so in order to solve the system, you'll need (at least) three equations. If we differentiate the first with respect to $y$ (and replacing $y$ with $x$), we get
$$f'(x) + xg(x) = 0. \tag{1}$$
If we differentiate the first equation instead with respect to $x$, we get
$$f'(x) + h(x) + xh'(x) - xg(x) = 0. \tag{2}$$
Differentiating the second equation with respect to $x$, we get
$$f'(x) + h'(x) - g(x) = 0. \tag{3}$$
Now we have three equations and some hope to solve the system. Adding $(1)$ and $(2)$ yields
$$2f'(x) + h(x) + xh'(x) = 0 \Longrightarrow \frac{d}{dx}\left(2f(x)+ xh(x)\right) = 0.$$
Hence $2f(x) + xh(x) = C$, i.e. $h(x) = \frac{C-2f(x)}{x}$. From $(1)$, $g(x) = -\frac{1}{x}f'(x)$. But from $(3)$, we also have that
$$g(x) = (f(x)+h(x))' = \frac{x(f(x)+xf'(x)-2f'(x))-(C+xf(x)-2f(x))}{x^2}$$
which simplifies to
$$g(x) = \frac{x^2f'(x)-2xf'(x)+ 2f(x)-C}{x^2}.$$
Setting the two expressions for $g$ equal to each other gives
$$-\frac{1}{x}f'(x) = \frac{x^2f'(x)-2xf'(x)+ 2f(x)-C}{x^2},$$
i.e.
$$-xf'(x) = x^2f'(x) -2xf'(x)+ 2f(x)-C.$$
So $f$ solves the differential equation
$$(x^2-x)f'(x) + 2f(x)-C = 0.$$
This can be solved by integrating factors. From this you can get $g$ and $h$. To get the constants of integration, you'll likely need to use your original equations. Of course you'll have to check that each of $f$, $g$ and $h$ are non-negative functions.