Given $X_1$ is $\Gamma(\alpha,1)$ distributed and $X_2$ is $\Gamma(\beta,1)$ distributed and set
$$Y=\frac{X_1}{X_1+X_2}.$$ The task is to show that $Y$ is $\operatorname{Beta}(\alpha,\beta)$ distributed.
I use the following for hint trying to solve the problem: set $Y_1=X_1$ and $Y_2=\frac{X_1}{X_1+X_2}$.
So I solve for $X_1=Y_1$ and $X_2=\frac{Y_1(1-Y_2)}{Y_2}$. By the transformation theorem, I need to find the Jacobian for $X_1$ and $X_2$, that is $|J| = -\frac{Y_1}{Y_2}$. Now from here I am quite confused. Should I use the probability density functions $\Gamma(a,1)$ and $\Gamma(b,1)$ and insert the expressions of $X_1$ and $X_2$ in then and multiply with the Jacobian? Because when I do so, I really get messed up calculations that does not make any sense. Would appreciate for help.
I assume $X_1,X_2$ are independent. As Saty suggested, it's an easier transformation of variables if you set
$$Y_1 = X_1+X_2\\ Y_2 = \dfrac{X_1}{X_1+X_2}.$$
Then $X_1 = Y_1Y_2$ and $X_2 = Y_1-Y_1Y_2$ and the Jacobian is
$$ J = \begin{vmatrix} y_2 & 1-y_2 \\ y_1 & -y_1 \\ \end{vmatrix} = -y_1. $$ Then,
\begin{eqnarray*} f_{Y_1,Y_2}(y_1,y_2) &=& f_{X_1,X_2}(x_1(y_1,y_2),\;x_2(y_1,y_2))\vert J\vert \\ &=& \dfrac{1}{\Gamma(\alpha)}(y_1y_2)^{\alpha-1}e^{-y_1y_2}\;\dfrac{1}{\Gamma(\beta)}y_1^{\beta-1}(1-y_2)^{\beta-1}e^{y_1y_2-y_1}y_1 \\ &=& \dfrac{1}{\Gamma(\alpha)\Gamma(\beta)}y_1^{\alpha+\beta-1}e^{-y_1}\;y_2^{\alpha-1}(1-y_2)^{\beta-1} \\ &=& \left( \dfrac{1}{\Gamma(\alpha+\beta)}y_1^{\alpha+\beta-1}e^{-y_1} \right) \left( \dfrac{1}{B(\alpha,\beta)}y_2^{\alpha-1}(1-y_2)^{\beta-1} \right) \quad\text{using $B(\alpha,\beta)=\dfrac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$.} \\ \end{eqnarray*}
Since $f_{Y_1,Y_2}(y_1,y_2)$ has form $f_{Y_1}(y_1)f_{Y_2}(y_2)$ and $f_{Y_2}(y_2)$ is a Beta density function with paramaters $\alpha,\beta,$ we are done.