Let $U \sim U([0, 1])$, be uniformly distributed on the interval $[0, 1]$. Let $X \sim U([0, U])$ and $Y \sim U([0, 1-U])$.
(a) Find the conditional density of $U$ given $Y$.
(b) Find the joint density of $X$ and $Y$.
For part (a), I tried to use Bayes rule in the following way:
$$f_{U | Y}\left(u | y\right) = \frac{f_{Y | U}\left(y | u\right) f_{U}\left(u\right)}{f_{Y}\left(y\right)}$$
Now, we know that $f_U(u) = 1$ for $0 \leq u \leq 1$ and 0 otherwise. The conditional density for $Y$ given that $U = u$ is $f_{Y | U}\left(y | u\right) = \frac{1}{1-u}$ for $0 \leq y \leq 1-u$. Additionally, the density of $Y$ can be given by:
$$ f_Y(y) = \int_{u = 0}^{u = 1}{f_{Y | U}\left(y | u\right) f_U(u) du} = \int_{0}^{1}{ \frac{1}{1-u} du}$$
However, this integral is divergent. So I am confused as to what could be done for this problem. I find that I run into similar divergence of integrals in part (b) as well.
You correctly state that $f_{Y\mid U}(y\mid u)=\frac1{1-u}$ for $0\le y\le1-u$, but then you use it as if it were $\frac1{1-u}$ everywhere. If you take into account that it’s $0$ outside that range, you get
$$ f_Y(y)=\int_0^1f_{Y\mid U}(y\mid u)f_U(u)\,\mathrm du=\int_0^{1-y}\frac1{1-u}\,\mathrm du=-\log y\;. $$