I'm physics student and not very good at proof. My mathematical physics textbook states the orthogonality of Bessel functions, http://mathworld.wolfram.com/BesselFunctionoftheFirstKind.html
basically same as the equation (53) in the linked page.
However, textbook says that there is a condition that $\nu>-1$ but doens't explain why. Wikipedia also states the same equation and condition but doesn't explain why. I can guess that it's related to the convergence of the integral but I'm not sure how to show it.
Can somebody help me?
Use the limiting form $J_{\nu}(az) \sim \left(\frac{az}{2}\right)^{\nu+1}/\Gamma(\nu+1).$ Then for $z$ near $0$ the integrand behaves like
$$z J_{\nu}(az)J_{\nu}(bz) \sim f(\nu,a,b) \left(\frac{z}{2}\right)^{2\nu+1}$$
where $f(\nu,a,b)$ is independent of $z$.
Therefore the integral converges if $2\nu + 1 > -1 $ or $\nu >-1.$