For an application, I was reading the wikipedia page on the Hankel transform and I was hoping somebody could clarify two things for me that I have not been able to find elsewhere as well:
1) Wikipedia defines the Hankel transform of a function $f(r)$ as $$ F_{\nu}(k) = \int_0^{\infty}f(r)J_{\nu}(kr)rdr, $$ with $J_{\nu}(kr)$ the Bessel function of the first kind of order $\nu \geq -1/2$. I was wondering why the order is restricted to be larger or equal than -1/2? Especially in the case of $\nu$ being an integer, one has $J_{-n}(x) = (-1)^n J_n(x)$, so I would say there should not be no problem in the definition.
2) One important property is the orthogonality condition $$ \int_0^{\infty}J_{\nu}(kr)J_{\nu}(k'r) rdr = \frac{\delta(k-k')}{k}, $$ for $k,k'>0$, with $\delta(k-k')$ the Dirac delta function. I was wondering if one can generalize this to something like $$ \int_0^{\infty}J_{\mu}(kr)J_{\nu}(k'r) rdr = \frac{\delta(k-k')\delta_{\mu\nu}}{k}, $$ because this would be really useful to me!
In my answer $n\geq 0$ is always an integer and $\nu\geq 0$ a non-integer real number.
1) $J_{-n}(z)$ behaves quite differently to $J_{-\nu}(z)$ near $z=0$. For example,
$$J_{-\nu}(z)\sim \frac{1}{\Gamma(1-\nu)}\left(\frac{2}{z}\right)^\nu,\qquad z\ll1$$
while
$$J_{-n}(z)\sim \frac{(-1)^n}{n!}\left(\frac{z}{2}\right)^n,\qquad z\ll 1.$$
Therefore, one has to be more restrictive on $f$ for $F_{-\nu}(k)$ to be well-defined when $-1<\nu<-\frac{1}{2}$, see http://dlmf.nist.gov/10.22#E77 for a discussion.
2) The closest I have seen to what you want are formulas (10.22.57) and (10.22.58) on NIST http://dlmf.nist.gov/10.22#E57 http://dlmf.nist.gov/10.22#E58. These are not orthogonality statements.