In the celebrated 8th edition (the latest) of I.S. Gradshteyn and I.M. Ryzhik, revised by D. Zwilinger and D. Moll (2015), on page 811 one reads formula 7.374.4 reproduced below:
$$\int_{-\infty}^\infty e^{-x^2} H_{2m+n}(ax)H_n(x) \,dx = \sqrt{\pi} \, 2^{-m+\frac{1}{2}} \frac{(2m+n)!}{m!}(a^2-1)^m a^n\qquad (1)$$
$m,n \in \mathbb{N}$, where $H_n$ is the so-called physicist's Hermite polynomial defined by $H_n(x) = (-1)^n e^{x^2}\frac{d^n}{dx^n}e^{-x^2}$.
The source of the formula is indicated as formula (21) on p. 291 in the well-known "Bateman project" reference book by Erdelyi's et al., Tables of Integral Transforms, vol. II. McGraw Hill, New York, 1954.
I checked that there was a misprint by Gradshteyn and al. in reproducing Erdelyi's original formula: they just replaced $e^{-\frac{1}{2} x^2}$ with $e^{-x^2}$ in the LHS integrand. Also Numerical experiments show both formulae to be wrong: for example using the following Mathematica code:
k[a_, c_, m_, n_] := NIntegrate[Exp[-c x^2] HermiteH[2 m + n, a x] HermiteH[n, x], {x, -Infinity, Infinity}]
k[0.01, 1, 3, 4] (* -0.171465 *)
k[0.01, 0.5, 3, 4] (* -901429 *)
l[a_, m_, n_] := Sqrt[Pi] 2^(-m + 1/2) Factorial[2 m + n]/Factorial[m] (a^2 - 1)^m a^n
l[0.01,3,4] (* -0.00189444 *)
I found that the correct formula in Gradshteyn et al. should be instead:
$$\int_{-\infty}^\infty e^{-x^2} H_{2m+n}(ax)H_n(x) \,dx = \sqrt{\pi} \, 2^{n} \frac{(2m+n)!}{m!}(a^2-1)^m a^n\qquad (2)$$
which is quite a difference from the published formula.
Numerical experiments confirm this:
p[a_, m_, n_] := Sqrt[Pi] 2^n Factorial[2 m + n]/Factorial[m] (a^2 - 1)^m a^n
p[0.01,3,4] (* -0.171465 = k[0.01, 1, 3, 4], as expected *)
I have seen no errata anywhere for this formula. Yet, notwithstanding the authoritative references, $(2)$ works and $(1)$ does not.
Question: Either prove the corrected version (2) of the formula or prove that I am wrong.
Let us denote:
$$I_{m,n}(a) = \int_{-\infty}^\infty e^{-x^2} H_{2m+n}(ax)H_n(x) \, dx$$
As
$$H_{2m+n}(ax) = (2m+n)! \sum_{k=0}^{\lfloor{m+\frac{n}{2}}\rfloor} \frac{(-1)^k}{k!\,(2m+n-2k)!} (2ax)^{2m+n-2k} \tag{3},$$
by integrating $(3)$, we obtain:
$$I_{m,n}(a) = (2m+n)! \sum_{k=0}^{\lfloor{m+\frac{n}{2}}\rfloor} \frac{(-1)^k}{k!\,(2m+n-2k)!} (2a)^{2m+n-2k} J_{2m+n-2k,n}(a) $$
in which
$$J_{r,n}(a) = \int_{-\infty}^\infty x^r e^{-x^2} H_n(x) dx $$
Now the integral $J_{r,n}(a) $ for $r$ positive integral can be obtained from the $n^{th}$ coefficient of the (finite) expansion of $x^{r}$ in a Hermite polynomial basis. The classic inverse expansion formula:
$$ x^r = \frac{r!}{2^r} \sum_{k=0}^{\lfloor{\frac{r}{2}}\rfloor}\frac{1}{k! (r-2k)!}H_{r-2k}(x)$$
therefore yields, for $r-2k=n$, considering the normalizing coefficient $2^n n! \sqrt{\pi}$:
\begin{align}J_{r,n}(a) &= \frac{r!}{2^{r}\,\frac{r-n}{2}!\,n!} \times 2^n n! \sqrt{\pi} \\ &= \frac{r!}{2^{r-n}\,\frac{r-n}{2}!} \sqrt{\pi} \\ \end{align} for $r \geq n$ and $J_{r,n}(a)=0$ otherwise, by an obvious argument on polynomial degrees.
Whence:
\begin{align} I_{m,n}(a) &= 2^n a^{2m+n} (2m+n)! \sqrt{\pi} \, \frac{1}{m!}\sum_{k=0}^{m} \frac{(-\frac{1}{a^2})^k}{k!\,} \frac{m!}{(m-k)!}\\ &= \sqrt{\pi} 2^n a^{2m+n} \frac{(2m+n)!}{m!} \, \frac{(a^2-1)^{m}}{a^{2m}} \quad \text{ by the binomial theorem}\\ &= \sqrt{\pi} 2^n \frac{(2m+n)!}{m!} \, (a^2-1)^{m} a^{n} \end{align}
which is $(2)$.