Let $X$ have a uniform distribution on $(0, 1)$, and given that $X = > x$, let the conditional distribution of $Y$ be uniform on $(0, 1/x)$.
(a) Find the joint pdf $f(x, y)$ and sketch the region where it is positive.
(b) Find $f_Y (y)$, the marginal pdf of $Y$ and sketch its graph.
(c) Compute $P(X > Y )$.
Given: $X\sim\text{unif}(0,1), \ X=x$ and $f_Y(y|x)\sim \text{unif}(0,1/x).$
(a): We are looking for $f(x,y)$. We have that
$$f_Y(y|x)=\frac{f(x,y)}{f_X(x)} \Longleftrightarrow f(y,x)=f_Y(x|y)f_X(x) \tag{1}$$
and $f_X(x)=1/(1-0)=1, \ f_Y(y|x)=1/(1/x-0)=x.$ So
$$f(x,y)=x\cdot1=x,$$
for $0\le x \le 1$ and $0\le y \le 1/x.$
The region is just everything below the graph of $y=1/x$, between $x=0$ and $x=1$.
(b): We have that
$$f_Y(y)=\int_0^1f(x,y) \ dx=\int_0^1x \ dx=\frac{1}{2}.$$
In the book, they answer that $f_Y(y)=1/2$ for $y\in[0,1]$ and $f_Y(y)=1/2y^2$ for $y\in[1,\infty].$ I don't understand how they get the bounds for $y$ and why there are two answers?
c): For continous random variables it follows that
$$P(X>Y)=P(X\geq Y)=1-P(X\leq Y).$$
Here I'm stuck, I don't know what I should replace $Y$ with.
(b): It may help to draw the region where $f(x,y)>0$. Look at the contraints which define this region: $$ 0\le x\le 1,0\le y\le 1/x $$ Note that the second constraint implies $x\le 1/y$. Therefore, we have $$ x\le 1\text{ and }x\le 1/y $$
Which of these two constraints is more restrictive? It depends on the value of $y$: if $y\le 1$, then the constraint $x\le 1$ is stronger, but if $y\ge 1$, the $x\le 1/y$ is. Therefore, the bounds on the integral $\int f(x,y)\,dx$ will depend on $y$. When $y\le 1$, you want $\int_0^1 f(x,y)\,dx$, and when $y\ge 1$, you want $\int_0^{1/y}f(x,y)\,dx$. That is where the two answers come from.
(c): You now need to integrate $f(x,y)$ over the region defined by the constraints $0\le x\le 1,0\le y\le 1/x$, with the additional constraint $x\ge y$. I suggest drawing this region, $R$ and using the equation of the boundary curves to determine the bounds for the integral $\iint_R f(x,y)\,dx\,dy$.