Suppose the following random variables have the following joint probability distribution: $$f(x,y)=\begin{cases}cx^{n-1}(y-x)^{m-1}e^{-y},0<x<y<\infty\\0, \text{else}\end{cases}$$ Find the value of $c$, $f_x$ and $f_y$.
I know that $$\int_0^{\infty}\int_0^ycx^{n-1}(y-x)^{m-1}e^{-y}dxdy=1$$
I'm in trouble in the caclulation of the integral. It is clear that it will be a gamma function at some point, but how can I find it?
Thanks for attention.
Recall that for $a, b > 0$, $$\int_{z=0}^1 z^{a-1} (1-z)^{b-1} \, dz = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}. \tag{1}$$ This is the usual beta function identity.
Then with a scaling transformation of the form $z = x/y$, $dz = \frac{1}{y} \, dx$, $(1)$ becomes $$\frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)} = \int_{x=0}^y (x/y)^{a-1}(1-x/y)^{b-1} \tfrac{1}{y} \, dx = \frac{1}{y^{a+b-1}} \int_{x=0}^y x^{a-1} (y-x)^{b-1} \, dx. \tag{2}$$
So $$\int_{y=0}^\infty \int_{x=0}^y c x^{n-1} (y-x)^{m-1} e^{-y} \, dx \, dy = \int_{y=0}^\infty c e^{-y} y^{a+b-1} \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)} \, dy. \tag{3}$$
I leave the remainder of the computation to you.