The joint probability density function of $X$ and $Y$ is given by $$f(x,y) = \begin{cases}c(y^2-x^2) e^{-y}, & -y \le x < +y, \ 0 \le y < \infty,\\ 0, & \text{otherwise}. \end{cases}$$
(a) Find $c$.
(b) Find the marginal densities of $X$ and $Y$.
(c) Find $\operatorname{E}[X]$.
I have some problems with the point (b)
(a) $$ \int_0^\infty\ \int_{-y}^y c(y^2-x^2)e^{-y}\,dy\,dx = 1 \Leftrightarrow c= \frac 1 8. $$ (b) I calculate the marginal density of $Y$ as $$ \int_{x=-y}^{x=+y} \frac 1 8 (y^2-x^2)e^{-y}\,dx = \frac 1 6 y^3 e^{-y}, $$ and the density of $X$ as $$ \int_{y=0}^\infty\ c(y^2-x^2)e^{-y}\,dy, $$ but there something wrong because the solution is different. Can someone help me to understand my mistake?
The marginal density of $Y$ is $$\int_{-y}^y c(y^2-x^2) e^{-y} \mathop{dx}=ce^{-y}(2y^3 - (2/3)y^3 ) = \frac{4c}{3} y^3 e^{-y}.$$
Integrating this over $y$ gives $$1=\int_0^\infty \frac{4c}{3} y^3 e^{-y} \mathop{dy} = \frac{4c}{3} \cdot 3! = 8c \implies c=1/8.$$
So, the marginal density of $Y$ is $\frac{1}{6} y^3 e^{-y}$ as you obtained.
The marginal density of $X$ is $$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy} .$$ The tricky part is the limits of integration, which comes from the condition $|x| \le y$. First, integration by parts twice gives \begin{align} \int_{|x|}^\infty y^2 e^{-y} \mathop{dy} &=[-y^2 e^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty ye^{-y} \mathop{dy}\\ &= x^2 e^{-|x|} + 2[-ye^{-y}]_{y=|x|}^\infty + 2\int_{|x|}^\infty e^{-y}\mathop{dy}\\ &= x^2 e^{-|x|} + 2|x|e^{-|x|} + 2e^{-|x|}\\ &= e^{-|x|}(|x|^2+2|x|+2). \end{align}
So, the marginal density of $X$ is $$\int_{|x|}^\infty c(y^2-x^2) e^{-y} \mathop{dy} = c(e^{-|x|}(|x|^2+2|x|+2) - x^2 e^{-|x|}) = \frac{1}{4} e^{-|x|}(|x|+1).$$
Edit (explanation for why the lower limit of integration is $|x|$):
In general, the marginal density of $X$ is $$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy}.$$ For a specific $x$, note that by definition $f(x,y)$ is zero if $|x|>y$; otherwise $f(x,y) = c(y^2-x^2)e^{-y}$. So $$f_X(x) = \int_{-\infty}^\infty f(x,y) \mathop{dy} = \int_{-\infty}^{|x|} f(x,y) \mathop{dy} + \int_{|x|}^\infty f(x,y) \mathop{dy} = 0 + \int_{|x|}^\infty c(y^2-x^2)e^{-y} \mathop{dy}.$$