I have been working a few hours on this particular problem. Please excuse my lack of formatting.
This is the question: Let $X$ and $Y$ be random variables with density function $f(x) = 2x$ on $[0, 1]$ and $0$ otherwise.
(a) Calculate the density function $X^2$.
(b) Using the convolution product, calculate the density of $X^2 + X$.
We have that $$ F_X(x)=2\int_0^xs\mathrm ds=x^2. $$ Hence, $F_{X^2}(x)=\Pr\{X^2\le x\}=F_{X}(\sqrt x)=x$ and the density function $f_{X^2}(x)=1$.
I'm not sure how to use the convolution product to calculate the density of $X^2+X$, but I suggest the following approach. Let us observe that $y=x^2+x$ is a monotone function when $0\le x\le 1$ and $x=1/2(\sqrt{4y+1}-1)$. Then we obtain the cumulative distribution function \begin{align*} F_{X^2+X}(x) &=\Pr\{X^2+X\le x\}\\ &=\Pr\{X\le1/2(\sqrt{4x+1}-1)\}\\ &=1/4(\sqrt{4x+1}-1)^2\\ &=1/2(2x+1-\sqrt{4x+1}) \end{align*} and the density function $$ f_{X^2+X}(x)=1-\frac1{\sqrt{4x+1}}. $$