On the triangle $0<x<1,0<y<x$ my random variable $R$ has density $p(x,y) = 1/x$.
I want the density $p_t:\mathbb{R}\to[0,\infty)$ of the transformation $t(R)$ where $t:\mathbb{R}^2\to\mathbb{R}$ is defined by $t(x,y)= y/x$. I thought I could obtain it like this
$$p_t(M)=\int_{\{(x,y)\ :\ 0\ <\ x\ <\ 1,\ 0\ <\ y\ <\ x,\ t(x,y)\ \in\ M\}}p(x,y)/\det\left( \begin{array}{cc} -\frac{y}{x^2} & \frac{1}{x} \\ \end{array} \right)\ dxdy$$
but the jacobian of the transformation is not rectangular so I can't take the determinant. What am I supposed to do?
If $X$ and $Y$ are random variables with joint density function $$f_{X,Y}(x,y) = \begin{cases} \frac 1x, &0 < x < 1, 0 < y < x,\\0, &\text{otherwise,}\end{cases}$$ and $Z = \frac YX$, then it is easier to find the density of $Z$ by computing its cumulative probability distribution function and then differentiating. Of course, if you are required to use the Jacobian method, then Did's hints are the way to go.
We have that for $0 < a < 1$, \begin{align}F_Z(a) = \int_{x=0}^1\int_{y=0}^{ax} f_{X,Y}(x,y) \,\mathrm dy \, \mathrm dx \end{align} where both sides can be differentiated with respect to $a$ before doing the computation of the double integral to get the density of $Z$ as a single integral with respect to $x$ (pretty much like finding the joint density of $X$ and $Z$ and then marginalizing with respect to $X$ as suggested in Did's hint).