UPD: I see the question wrongly, thanks for you guys.
I use integration by parts to calculate the left side of the equation. Like $$\int x^2e^{-x^2} dx$$ $$= -x^2e^{-x} - \int 2x(-e^{-x}) dx$$ $$= -x^2e^{-x} - 2xe^{-x} - 2e^{-x} + C$$
So $\int_0^{\infty} x^2e^{-x^2} dx = \lim_{t \to \infty} ((-t^2e^{-t} - 2xe^{-t} - 2e^{-t}) - (-2)) = 2$
But I don't know how to calculate the right side of the equation.
Can anyone give me some tips? Thanks in advance!
You wrote $e^{-x^2}$ as $e^{-x}$ in your second line, so your working is wrong. Instead:
\begin{align*} \int_0^\infty x^2e^{-x^2} \mathrm{d}x &= \int_0^\infty -\frac{x}{2}(-2xe^{-x^2}) \mathrm{d}x \\ &= \left[-\frac{x}{2}e^{-x^2}\right]_{x = 0}^{x = \infty} - \int_0^\infty -\frac{1}{2}e^{-x^2} \mathrm{d}x \\ &= \frac{1}{2} \int_0^\infty e^{-x^2} \mathrm{d}x \end{align*} Several details were omitted, and I'll leave you to figure them out.