Calculate the integral $\int_0^{\infty} x^{11} e^{-x^3}dx$.
I wanted to approach this using the reduction formula. Here's what I have tried $$\int x^2e^{-x^3}dx = -\frac{1}{3}e^{-x^3}$$
Then extending this to $n$ $$I_n = \int_0^{\infty}x^ne^{-x^3}dx $$
Using substitution I can get
$$u = x^{n-2}; du = (n-2)x^{n-3}; dv = x^2e^{-x^3}; v = -\frac{1}{3}e^{-x^3}$$
Plugging this in $$I_n = \int_0^{\infty}x^ne^{-x^3}dx = \left[-\frac{x^{n-2}}{3}e^{-x^3} \right]_0^{\infty}+\frac{(n-3)}{3} \int_0^{\infty}x^{n-3}e^{-x^3}dx $$
$$\implies I_n = \int_0^{\infty}x^ne^{-x^3}dx =\frac{(n-3)}{3} \int_0^{\infty}x^{n-3}e^{-x^3}dx$$ $$=\frac{n-3}{3}I_{n-3}$$
Then starting from $n=11$ I get $$\frac{8}{3} \cdot \frac{5}{3} \cdot\frac{2}{3} \int_0^{\infty} x^2e^{-x^3}dx$$
However this does not produce the answer which is $2$. How should I go about the reduction formula for this?
Rather than trying to do a reduction formula on the original integrand, first perform the substitution $$y = x^3, \quad dy = 3x^2 \, dx.$$ Then for an integrand of the form $$f_n(x) = x^{3n+2} e^{-x^3}$$ we have $$\int f_n(x) \, dx = \frac{1}{3}\int y^n e^{-y} \, dy.$$ This integral is now amenable to integration by parts with the usual choice $$u = y^n, \quad du = ny^{n-1} \, dy, \\ dv = e^{-y} \, dy, \quad v = -e^{-y}:$$
$$I_n(y) = \int y^n e^{-y} \, dy = -y^n e^{-y} + n \int y^{n-1} e^{-y} \, dy = -y^n e^{-y} + n I_{n-1}(y).$$
If we undo the initial substitution, this gives
$$\int x^{3n+2} e^{-x^3} \, dx = -\frac{1}{3}x^{3n} e^{-x^3} + \frac{n}{3} \int x^{3n-1} e^{-x^3} \, dx.$$
However, in your case, we simply have $n = 3$, and
$$\begin{align} I_3(y) &= -y^3 e^{-y} + 3 I_2(y) \\ &= -y^3 e^{-y} - 3y^2 e^{-y} + 6 I_1(y) \\ &= -y^3 e^{-y} - 3y^2 e^{-y} - 6y e^{-y} + 6 I_0(y) \\ &= -e^{-y} (y^3 + 3y^2 + 6y + 6), \end{align}$$
and in terms of $x$, this gives
$$\int f_3(x) \, dx = -\frac{1}{3}e^{-x^3}(x^9 + 3x^6 + 6x^3 + 6) + C.$$
So the definite integral is $$\int_{x=0}^\infty f_3(x) \, dx = 0 + \frac{6}{3} = 2.$$