I'm attempting to find a closed form expression for $$\int_{-\infty}^{\infty}e^{-\frac{x^2\left(1+\lambda^2\right)}{2}}H_{n}(x)H_m(\lambda x)dx$$
where $H_n(x)$ are the physicist's hermite polynomials, but haven't had any luck. Anyone know of a way to compute this?
On
There is a more compact closed-form solution expressed with hypergeometric functions, using known Laplace transforms. The method used can be recycled to find a closed-form solution for other integrals of the same type; this is why I am adding this answer despite the fact that it is an old question.
Noting that for $k \in \mathbb{N}_0$:
$$H_{2k}(x) = (-1)^k 2^{2k}\, k! \, L_{k}^{-\frac{1}{2}}(x^2) $$ $$H_{2k+1}(x) = (-1)^k 2^{2k+1}\, k! \, x\, L_{k}^{\frac{1}{2}}(x^2), $$
in which $L_k^\alpha$ is a generalized Laguerre polynomial; for even values of the indices of the posted integral $I_{n,m}$, an obvious change of variables gives:
\begin{align} I_{2k,2p} &= (-1)^{k+p}2^{2(k+p)}k!\,p! \int_{0}^\infty u^{-\frac{1}{2}} e^{-\frac{u}{2} (1+\lambda^2)} L_k^{-\frac{1}{2}}(u)L_p^{-\frac{1}{2}}(\lambda^2 u)\,du \\ &= (-1)^{k+p}2^{2(k+p)}k!\,p!\; \mathscr{L}\left(u^{-\frac{1}{2}} L_k^{-\frac{1}{2}}(u)L_p^{-\frac{1}{2}}(\lambda^2 u); \frac{1}{2}(1+\lambda^2)\right) \tag{1} \end{align} in which $\mathscr{L}(f; y)$ is the standard Laplace transform of function $f$ taken at point $y$. This Laplace transform is a classic that is to be found in published tables. The general formula is [Gradshtein 2015, eq. 7.414.4 p.817] :
$$\mathscr{L}\left(u^\alpha L_n^\alpha(b\, u)L_m^\alpha(c\,u); y\right) = \frac{\Gamma(m+n+\alpha+1)(y-b)^n(y-c)^m}{m! \, n! \, y^{m+n+\alpha+1}}{_2}F{_1}\left(-m, -n; -m-n-\alpha; \frac{y(y-b-c)}{(y-b)(y-c)}\right), \tag{2}$$
where ${_2}F_{1}$ denotes the Gauss hypergeometric function, under the provisions that $\Re \alpha > -1$ and that $\Re y > 0$.
Setting in $(2)$: $\alpha = -\frac{1}{2};\,b= 1;\, c= \lambda^2; y =\frac{1}{2}(1+\lambda^2)$, all provisions are obviously satisfied, for $\lambda$ real. Note that the question does not make it specific that $\lambda$ be real. If it were a complex number instead, there is good reason to believe that there would be no solutions for $\Re \lambda^2 \leq -1$, considering the validity domain of the Laplace transform formula, a non-trivial reservation that it would be harder to make following other approaches.
With these reservations, we have:
$$\mathscr{L}\left(u^{-\frac{1}{2}} L_k^{-\frac{1}{2}}(u)L_p^{-\frac{1}{2}}(\lambda^2\, u); \frac{1}{2}(1+\lambda^2)\right) = (-1)^k\frac{\sqrt{2}\Gamma(p+k+\frac{1}{2})(1-\lambda^2)^{p+k}}{p! \, k! \, (1+\lambda^2)^{p+k+\frac{1}{2}}}{_2}F{_1}\left(-p, -k; -p-k+\frac{1}{2}; \left(\frac{1+\lambda^2}{1-\lambda^2}\right)^2\right) \tag{3}$$
which gives the solution to the question, for even indices:
\begin{align} I_{2k,2p} &= (-1)^{p}2^{2(k+p)+\frac{1}{2}} \frac{\Gamma(p+k+\frac{1}{2})(1-\lambda^2)^{p+k}}{(1+\lambda^2)^{p+k+\frac{1}{2}}}{_2}F{_1}\left(-p, -k; -p-k+\frac{1}{2}; \left(\frac{1+\lambda^2}{1-\lambda^2}\right)^2\right) \tag{4} \end{align}
This formula can also be recast using only factorials, by the Legendre duplication formula, as follows:
\begin{align} I_{2k,2p} &= (-1)^{p} 2\sqrt{2 \pi} \frac{(2(p+k)-1)! \, (1-\lambda^2)^{p+k}}{(p+k-1)!\,(1+\lambda^2)^{p+k+\frac{1}{2}}}{_2}F{_1}\left(-p, -k; -p-k+\frac{1}{2}; \left(\frac{1+\lambda^2}{ 1-\lambda^2}\right)^2\right) \tag{5} \end{align}
For odd indices of the Hermite polynomials, $(1)$ becomes
\begin{align} I_{2k+1,2p+1} &= (-1)^{k+p}2^{2(k+p+1)}k!\,p!\; \lambda\; \mathscr{L}\left(u^{\frac{1}{2}} L_k^{\frac{1}{2}}(u)L_p^{\frac{1}{2}}(\lambda^2 u); \frac{1}{2}(1+\lambda^2)\right) \tag{6}, \end{align}
as we now take $\alpha = \frac{1}{2}$, so that $(4)$ becomes
\begin{align} I_{2k+1,2p+1} &= (-1)^{p}2^{2(k+p+1)+\frac{3}{2}} \lambda \;\frac{\Gamma(p+k+\frac{3}{2})(1-\lambda^2)^{p+k}}{(1+\lambda^2)^{p+k+\frac{3}{2}}}{_2}F{_1}\left(-p, -k; -p-k-\frac{1}{2}; \left(\frac{1+\lambda^2}{1-\lambda^2}\right)^2\right) \tag{7} \end{align}
Note that the Gauss function has a negative integer first parameter and so can only be a polynomial, which explains the polynomial form of Tom Davis's solution. The above solution has been numerically tested using Mathematica 14.
References
Gradshtein, I. S. (Izrail Solomonovich), rev. Daniel Zwillinger [Tablitsy integralov, summ, riadov i proizvedenii. English] Table of integrals, series, and products. – Eighth edition, Elsevier, 2015.
On
The integral when $n=m$ is $$ I_{nn} = 2^{2n}\sqrt{2\pi}\left(n!\right)^{2} \frac{\lambda^n}{{\left(\lambda^{2} + 1\right)^{n + \frac{1}{2}}}} {\sum_{k=0}^{\left \lfloor \frac{n}{2} \right \rfloor} \frac{\left(-1\right)^{k} }{2^{4k} (k!)^{2} \left(n-2k \right)!}}\left(\frac{{\lambda^{2} - 1}}{\lambda}\right)^{2k}. $$
The integral is zero whenever $n$ and $m$ have opposite parity. When $n\ge m$, define $s=\frac{n-m}{2}$ (which is guaranteed to be an integer) and the integral becomes $$ I_{nm}=2^{2m}\sqrt{2\pi}m! n!\frac{\lambda^{m}{\left(1-\lambda^{2} \right)}^{s} }{{\left(\lambda^{2} + 1\right)}^{m + s + \frac{1}{2}}}{\sum_{l=0}^{\left \lfloor \frac{m}{2} \right \rfloor} \frac{\left(-1\right)^{l} }{2^{4l} \left(l + s\right)! l! \left(m-2l\right)!}\left(\frac{{\lambda^{2} - 1}}{\lambda}\right)^{2l}} $$ The general case is $$ I_{nm}=2^{2m}\sqrt{2\pi}m! n!\frac{\lambda^{\operatorname{min}(n,m)}{\left(1-\lambda^{2} \right)}^{s}}{{\left(\lambda^{2} + 1\right)}^{\operatorname{max}(n,m) + \frac{1}{2}}}{\sum_{l=0}^{\left \lfloor \frac{\operatorname{min}(n,m)}{2} \right \rfloor} \frac{\left(-1\right)^{l} }{2^{4l} \left(l + s\right)! l! \left(\operatorname{min}(n,m)-2l\right)!}\left(\frac{{\lambda^{2} - 1}}{\lambda}\right)^{2l}} $$
In order to derive this result, first change to probabilists Hermite polynomials $$ H_n(x)=2^{\frac{n}{2}}\operatorname{He}_n(\sqrt{2}x) $$ so the integral becomes $$ I_{nm} = 2^{\frac{n+m}{2}}\int_{-\infty}^\infty e^{-\frac{x^2}{2}(1+\lambda^2)}\operatorname{He}_n(\sqrt{2}x)\operatorname{He}_m(\lambda\sqrt{2}x)dx. $$ Change the integration variable in order to recover the probabilists weighting function $y=x\sqrt{1+\lambda^2}$ $$ I_{nm} = \frac{2^{\frac{n+m}{2}}}{\sqrt{1+\lambda^2}} \int_{-\infty}^\infty e^{-\frac{y^2}{2}} \operatorname{He}_n\left(y\sqrt{\frac{2}{1+\lambda^2}}\right)\operatorname{He}_m\left(y\sqrt{\frac{2\lambda^2}{1+\lambda^2}}\right)dy. $$
Use the scaled Hermite polynomial on both polynomials $$ \operatorname{He}_n(\gamma x) = n!\sum_{k=0}^{\left \lfloor \frac{n}{2} \right \rfloor}\frac{1}{2^kk!(n-2k)!}\gamma^{n-2k}\left(\gamma^2-1\right)^k H_{n-2k}(x) $$ leads to a very long expression, from which all three cases ($n=m$, $n\ge m$, $n\le m$) obtain $$ I_{nm} = \frac{2^{\frac{n+m}{2}}}{\sqrt{1+\lambda^2}}n!m!\sum_{k=0}^{\left\lfloor\frac{n}{2}\right\rfloor}\sum_{l=0}^{\left\lfloor\frac{m}{2}\right\rfloor}\frac{(-1)^k\left(\sqrt{\frac{2}{1+\lambda^2}}\right)^{n-2k+m-2l}\lambda^{m-2l}\left(\frac{\lambda^2-1}{\lambda^2+1}\right)^{k+l}}{2^kk!(n-2k)!2^ll!(m-2l)!}\\ \times\int_{-\infty}^\infty\operatorname{He}_{n-2k}(x)\operatorname{He}_{m-2l}(x)e^{-\frac{x^2}{2}}dx. $$
Orthogonality will constrain one of the sums -- always choose the sum associated with the maximum of $n,m$. Take the case $n\ge m$, as previously stated, the parity of $n$ and $m$ has to be the same, otherwise the integrand is odd and the integral vanishes. Therefore write $n=m+2s$ for $s\in \mathbb{Z}$. The orthogonality constraint can be written $k=l+s$ and, after algebraic manipulation, the result above obtains.
Denoting the integral by $I_{m,n}$, write the generating function \begin{align*}I(s,t)=&\sum_{m=0}^{\infty}\sum_{n=0}^{\infty}I_{m,n}\frac{s^m}{m!}\frac{t^n}{n!}=\\ =&\int_{-\infty}^{\infty}e^{-\frac{x^2(1+\lambda^2)}{2}} \underbrace{\left(\sum_{m=0}^{\infty}H_m(\lambda x)\frac{s^m}{m!}\right)}_{e^{2\lambda x s-s^2}} \underbrace{\left(\sum_{n=0}^{\infty}H_n( x)\frac{t^n}{n!}\right)}_{e^{2xt-t^2}}dx=\\ =&e^{-s^2-t^2}\int_{-\infty}^{\infty}e^{-\frac{x^2(1+\lambda^2)}{2}+2x(\lambda s+t)}dx=\\ =&\exp\left\{\frac{2(\lambda s+t)^2}{1+\lambda^2}-s^2-t^2\right\}\sqrt{\frac{2\pi}{1+\lambda^2}}. \end{align*} The rest should be clear. Of course, $I_{m,n}=0$ if $m,n$ are of different parity.