If $X_i$, $i=1,2$, are independent $gamma(\alpha_i,1)$ random variables. Find the joint distribution of $X_1/(X_1+X_2)$ and $X_2/(X_1+X_2)$. I am trying to use transformation method to solve it.
Let $U=X_1/(X_1+X_2)$ and $V=X_2/(X_1+X_2)$, but I cannot represent $X_1=g_1(U,V)$ or $X_2=g_2(U,V)$
As I said in my comments, $W = \frac{X_1}{X_1+X_2}$ and $Z = \frac{X_2}{X_1+X_2}$ do not have a joint density function (joint pdf), and all the probability mass lies on the line segments with end points $(0,1)$ and $(1,0)$. You can work out the joint distribution function (joint CDF) $F_{W,Z}(w,z)$ (which, incidentally, is what the question asks for) quite easily. Some values can be put down by inspection:
$$F_{W,Z}(w,z) = \begin{cases} 0, & w \leq 0 ~\text{or}~ z \leq 0 ~\text{or}~ w+z \leq 1,\\ 1, & w \geq 1 ~\text{and}~ z \geq 1. \end{cases}$$
For $0 < w < 1, z > 1$, \begin{align} F_{W,Z}(w,z) &= P\{W \leq w, Z \leq z\}\\ &= P\left\{\frac{X_1}{X_1+X_2} \leq w, \frac{X_2}{X_1+X_2} \leq z\right\}\\ &= P\left\{\frac{X_1}{X_1+X_2} \leq w\right\}\\ \end{align} and you say you know how to compute that last probability since you say in a comment on your question that "It is not hard to find the pdf of one of them".
Similarly, for $w > 1, 0 < z < 1$, $$F_{W,Z}(w,z) = P\left\{\frac{X_2}{X_1+X_2} \leq z\right\} = P\left\{1-z \leq \frac{X_1}{X_1+X_2}\right\}$$ which you say you can compute. Finally, for the region $0 < w,z < 1, w+z > 1$, we have that $$F_{W,Z}(w,z) = P\left\{1 - z \leq \frac{X_1}{X_1+X_2} \leq w\right\}$$ which also can be computed. Thus, we have found the joint CDF for $W$ and $Z$. Note that only in the last case above is $F_{W,Z}(w,z)$ a function of both $w$ and $z$, but even here, $\displaystyle \frac{\partial^2F_{W,Z}(w,z)}{\partial w\partial z}$ is always $0$ thus adding credibility to the claim that $W$ and $Z$ do not have a joint density function.