Let $X$ and $Y$ be random variables with probability density $f(x, y)=12xy(1-y)$ if $0<x<1, 0<y<1$. Find the joint probability density of $U=XY^2$ and $V=Y$. Find the marginal density of $U$.
Attempt:
With what I have calculated, I have to $$f_{U, V}(u, v)=\frac{12u(1-v)}{v}\cdot \frac{1}{|v^2|} \text{ if $0<\frac{u}{v^2}<1, 0<v<1$}$$
But since $0<v<1$, we can remove the absolute value. Thus we have $$f_{U, V}(u, v)=\frac{12u(1-v)}{v^3} \text{ if $0<\frac{u}{v^2}<1, 0<v<1$}$$
Is this okay?
The next thing is to "solve" the domain (support) $0<\frac{u}{v^2}<1, 0<v<1$, and I believe that the above inequalities imply that $v\neq 0$, $u>0$.
You correctly obtained,
$ \displaystyle f_{U, V}(u, v)=\frac{12u(1-v)}{v^3} \text{ if $0<\frac{u}{v^2}<1, 0<v<1$}$
In fact you also found the support. The support is simply $ ~ 0 \lt u \lt v^2, 0 \lt v \lt 1$. You can also write it as $\sqrt u \lt v \lt 1, 0 \lt u \lt 1$. Please see the shaded region below.
So for marginal density of $U$,
$ \displaystyle f_U(u) = \int_{\sqrt u}^1 \frac{12u(1-v)}{v^3} ~ dv$
$ \implies \displaystyle f_U(u) = 6 - 12 \sqrt u + 6 u, 0 \lt u \lt 1$