Suppose $Y_1$ and $Y_2$ are two random variable have joint probability density function $$ f(y_1,y_2)= \begin{cases} 4y_1y_2&0<y_1<1,0<y_2<1\\ 0&\text{for other }x \end{cases}. $$ If $Y_1=X_1^2$ and $Y_2=X_1X_2$, determine joint p.d.f. of $X_1$ and $X_2$.
I use jacobian transformation, so I find the Jacobian determinant as below. $$ |J|=\left\vert \begin{matrix} \dfrac{\partial Y_1}{\partial X_1}&\dfrac{\partial Y_1}{\partial X_2}\\ \dfrac{\partial Y_2}{\partial X_1}&\dfrac{\partial Y_2}{\partial X_2}\\ \end{matrix} \right\vert = \left\vert \begin{matrix} 2X_1&0\\ X_2&X_1\\ \end{matrix} \right\vert = 2X_1^2. $$
Now, I find the joint p.d.f. as below. \begin{eqnarray} g(x_1,x_2)&=&f(y_1,y_2)\vert J\vert\\ &=& f\left(x_1^2,x_1x_2\right)\vert J\vert\\ &=& 4(x_1^2)(x_1x_2)(2x_1^2)\\ &=& 8x_1^5 x_2 \end{eqnarray}
After that, I determine the domain of $x_1$ and $x_2$.
$Y_1=X_1^2$ imply $X_1=\sqrt{Y_1}$ and $Y_2=X_1X_2$ imply $X_2=\dfrac{Y_2}{\sqrt{Y_1}}$.
$X_1=\sqrt{Y_1}$ and $0<Y_1<1$, it's clear that $0<X_1<1$. Now I'm confused to determining the range of $Y_2$.
$X_2=\dfrac{Y_2}{\sqrt{Y_1}}$, the denominator range is $0$ into $1$. the numerator range is $0$ into $1$. If numerator $>$ denominator, the range of $X_2$ is $>1$. But if denominator $>$ numerator, the range of $X_2$ is $0$ into $1$. So, the range of $X_2$ is $X_2>0$. Is it right?
Based on the result of joint pdf of $X_1$ and $X_2$, $g(x_1,x_2)=8x_1^5 x_2$, the double integral of $g(x_1,x_2)$ is $\infty$, cannot equal 1. So, the range of $X_2$ are wrong.
So how to determine the range of $X_1$ and $X_2$ on this problem?
The domain is ${\quad\{\langle x_1,x_2\rangle: (0\leq x_1^2\leq 1)\land (0\leq x_1x_2\leq 1)\}\\={\{\langle x_1,x_2\rangle: ({{(-1\leq x_1<0)\land(-1/x_1\leq x_2\leq 0))}\lor{((0\leq x_1\leq 1)\land(0\leq x_2\leq 1/x_1)}})\}}}$