I'm interested in the integral
$$ \int_0^{2 \pi} dx \ H_\ell (\cos x) H_m (\sin x) e^{ i n x} = 0 $$
I tried expressing the polynomials as contour integrals but got stuck. Maybe one can obtain some recursion relation?
Even if we can't do the integral, how would you prove that it vanishes if $n > \ell +m$?
This is actually pretty simple with contour integration - in fact, barely any calculation at all. Rewrite the integral as
$$-i \oint_{|z|=1} dz \, H_{\ell} \left (\frac{z+z^{-1}}{2} \right ) \, H_{m} \left (\frac{z-z^{-1}}{i 2} \right ) z^{n-1}$$
The Hermite $H_{\ell}$ has maximum degree $\ell$, so the highest-order term in $z^{-1}$ is $z^{-\ell}$, and similarly for $H_m$ is $z^{-m}$. Thus, is $n \gt \ell+m$ then the integrand is a polynomial and the integral is therefore zero by Cauchy's theorem.