We have two r.v.s, $X$ and $Y$, both have density $3x^2$ where $x$ in range $(0,1)$. we want the density function of $Z=XY$.
By the product distribution equation and the fact that $X,Y>0$ we have: $$f_{Z}(z)=\int_0^13x^2\cdot 3\left(\frac{z}{x}\right)^2\frac{1}{x}dx$$ But wolfram alpha told me that thing does not converge! I think I did everything right and I haven't missed anything, I ignored the absolute value on $\frac{1}{|x|}$ because $x$ is strictly larger than $0$. I don't know what's happening, please help me out! thank you!
You have missed the domain of your probability density functions.
I found this formula for evaluating the product distribution from Wikipedia:
$$f_{Z}(z)=\int _{{-\infty }}^{{\infty }}f_{X}\left(x\right)f_{Y}\left(z/x\right){\frac {1}{|x|}}\,dx.$$
Therefore for your case, we have $$ \begin{align} f_{Z}(z) &=\int _{{-\infty }}^{{\infty }} 3 x^2\cdot \mathbb{1}_{x \in (0, 1)}\cdot 3 \left(\frac{z}{x}\right)^2 \cdot \mathbb{1}_{z/x \in (0, 1)} \cdot {\frac {1}{x}}\,dx\\ &= \begin{cases} \displaystyle \int _{{z }}^{{1 }} 3 x^2\cdot 3 \left(\frac{z}{x}\right)^2 \cdot {\frac {1}{x}}\,dx & \text{if $z < 1$}\\ \displaystyle 0& \text{if $z \geq 1$} \end{cases}\\ &= 9z^2 \ln \frac1z \cdot \mathbb{1}_{z \in (0, 1)}, \end{align} $$ where $\mathbb{1}_{A}$ is the indicator function with the definition $$\mathbb{1}_{A} = \begin{cases}1 &\text{if $A$ is true}\\0 &\text{otherwise}.\end{cases}$$