Given joint PDF $f(x, y) = 8xy\mathbf 1_D(x, y)$, where $D = \{(x, y) \in \mathbb R^2 \mid 0 < x < y < 1\}$, calculate PDF of $Z = \max\{|X|, |Y|\}$.
I don't know what I'm doing wrong, but I observed that $|X| = X$ and $|Y| = Y$, since $X$ and $Y$ have non-zero density on $D$, which is just comprised by points with positive coordinates.
Then I calculated the densities of $X$ and $Y$: $$\begin{align*} f_X(x) &= \int_x^1 8xy\mathbf 1_D(x, y)\,\mathrm dy = 4x(1-x^2)\mathbf 1_{(0, 1)}(x) \\ f_Y(y) &= \int_0^y 8xy\mathbf 1_D(x, y)\,\mathrm dx = 4y^3\mathbf 1_{(0, 1)}(y) \end{align*}$$ from which I found the CDFs (with $x \in (0, 1)$ and $y \in (0, 1)$): $$\begin{align*} F_X(x) &= \int_0^x f_X(s)\,\mathrm ds = 2x^2 - x^4\\ F_Y(y) &= \int_0^y f_Y(s)\,\mathrm ds = y^4 \end{align*}$$
Hence, for $z \in (0, 1)$, $$F_Z(z) = \mathbb P[X \le z]\mathbb P[Y \le z] = z^4(2z^2 - z^4)$$ from which, using $f_Z(z) = \frac{\mathrm dF_Z(z)}{\mathrm dz}$, $$\boxed{f_Z(z) = (12z^5 - 8z^7)\mathbf 1_{(0, 1)}(z)}$$
Solution says $$\boxed{f_Z(z) = 4z^3\mathbf 1_{(0, 1)}(z)}$$
Where did I go wrong?
Because the domain of the joint pdf is $D = \{(x, y) \in \mathbb R^2 \mid 0 < x < y < 1\}$, the maximum of $X$ and $Y$ is always $Y$. That is, we need $Y$'s marginal density:
$$f_{\max(X,Y)}(y)=f_Y(y)=8y\int_0^yx\ dx=4y^3$$
over $y\in[0,1]$, $0$ otherwise.
So, your calculations derailed when you did not stop after getting $f_Y$.