While solving this particular definite integral: $$\int_0^\infty x^2 e^{-2ax} dx$$
I could evaluate the indefinite integral as: $$- \frac{e^{-2ax} (2a^2x^2 + 2ax + 1)}{4a^3}$$
But I fall into this problem of $0 * \infty$ upon putting the limits. What's the correct way to evaluate this integral?
As mentioned in the comments and answers, it is clear that you must have $a > 0$ for the integral to converge. Now, if you want to evaluate the integral as you suggest, separate the terms \begin{align*} - \frac{e^{-2ax} (2a^2x^2 + 2ax + 1)}{4a^3} \Big|_{0}^{\infty} & = \left(\lim_{x \to \infty} - \frac{e^{-2ax} (2a^2x^2 + 2ax + 1)}{4 a^3} \right) + \frac{1}{4a^{3}} \\ & = -\left(\lim_{x \to \infty} \frac{e^{-2ax}2a^2x^2}{4 a^3} + \frac{2e^{-2ax}ax}{{4 a^3} } + \frac{e^{-2ax}}{4 a^3} \right) + \frac{1}{4a^{3}} \\ & = -\left(\lim_{x \to \infty} \frac{x^2}{2 e^{2ax}a} + \frac{x}{{2e^{2ax} a^2} } + \frac{1}{4e^{2ax} a^3} \right) + \frac{1}{4a^{3}} \\ & = -(0 +0 + 0) + \frac{1}{4a^3} \\ & = \frac{1}{4a^3} \end{align*} in other case you cand use Gamma function as $$ \int_{0}^{\infty} x^{\alpha-1} e^{-\beta x} = \frac{\Gamma(\alpha)}{\beta^{\alpha}} $$