I have two independent random variables:
- $X$ is Beta($a, b$), $x \in [0, 1]$
- $Y$ is Beta($a+b, \gamma$), $y \in [0, 1]$
$$f(x,y) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} x^{a-1}(1-x)^{b-1}\frac{\Gamma(a+b+\gamma)}{\Gamma(a+b)\Gamma(\gamma)}y^{a+b-1} (1-y)^{\gamma-1}$$
I want to perform the transformations for $U = XY$, $V=Y$. $$U=XY \rightarrow U=XY \rightarrow X=U/V, \qquad u \in [0,1]$$ $$V=Y \rightarrow Y=V, \qquad v\in[0,1]$$
$$\det{J}= \frac{1}{v}*1-0=\frac{1}{v}$$ So I have,
$$f(u, v) = \frac{\Gamma(a+b+\gamma)}{\Gamma(a+b)\Gamma(\gamma)} (u/v)^{a-1}(1-(u/v))^{b-1}v^{a+b-1} (1-v)^{\gamma-1} * \frac{1}{v}, \qquad 0 \le u \le v \le 1$$
I want to find the marginal distribution of $u$.
$$f_{U}(u) = \frac{\Gamma(a+b+\gamma)}{\Gamma(a)\Gamma(b)\Gamma(\gamma)} u^{a-1} \int^1_{u} (1-(u/v))^{b-1}v^{a+b-1} (1-v)^{\gamma-1} \frac{1}{v} dv$$
QUESTION: However, I am unsure how to go about this integral. How can I proceed with this integration find the marginal distribution for $U$?
Suppose $X\sim \text{Beta}(a,b)$ and $Y\sim \text{Beta}(c,d)$ are independently distributed with $c=a+b$.
Then $(X,Y)$ has density $$f_{X,Y}(x,y)\propto x^{a-1}(1-x)^{b-1}y^{c-1}(1-y)^{d-1}\mathbf1_{0<x,y<1}\,,$$ where $a,b,c,d>0$.
Transforming $(X,Y)\mapsto (U,V)=(XY,Y)$, density of $(U,V)$ is
\begin{align} f_{U,V}(u,v)&=f_{X,Y}\left(\frac{u}{v},v\right)\cdot\frac1v \\&\propto u^{a-1}(v-u)^{b-1}(1-v)^{d-1}\mathbf1_{0<u<v<1} \end{align}
Pdf of $U$ is therefore
$$f_U(u)\propto u^{a-1}\int_u^1 (v-u)^{b-1}(1-v)^{d-1}\,dv\,\mathbf1_{0<u<1}$$
Now you can turn this integral into a Beta function if you change variables $v\mapsto t$ such that $t=0$ when $v=u$ and $t=1$ when $v=1$. This yields the relation $v-u=t(1-u)$.
On simplification you have another Beta distribution:
$$f_U(u)\propto u^{a-1}(1-u)^{b+d-1}\,\mathbf1_{0<u<1}$$
I have ignored the normalizing constants here since these are all guaranteed to be valid densities. You can of course verify that you end up with the correct constant.