What is the basis for functions on the Hilbert space $L^2([0,\infty), \frac{dx}{x})$. I am studying the Mellin transform and I'm trying to understand the role of the functions $n^{it} = e^{it \, \log n }$. These functions are orthonormal on $\mathbb{R}^+$ with the measure $d\mu = \frac{dx}{x} = d(\log x)$.
$$ \int_0^\infty m^{it}\, n^{-it}\,\frac{dt}{t} =\left\{\begin{array}{cc} 1 & \text{if }m=n \\ 0 & \text{if }m\neq n\end{array}\right. \leftrightarrow \int_0^\infty e^{imx}\, e^{-inx}\,dx =\left\{\begin{array}{cc} 1 & \text{if }m=n \\ 0 & \text{if }m\neq n\end{array}\right. $$ On the right side, I have written orthogonality relations for another Hilbert space $L^2(\mathbb{R}, dx)$. Aren't these both related by the map $t = e^x$ or the other way $x = e^t$ but I don't see that $e^{imx}$ and $m^{it}$ are related by the same substitution.
Until I look for a more official source, Wikipedia's discussion on the Mellin transform has this bit to say about the isometry $L^2(\mathbb{R}, dx)$ and $L^2(\mathbb{R}^\times , \frac{dx}{x})$.
\begin{eqnarray} \big[\mathcal{M}f\big](s) &=& \int_0^\infty x^s \, f(s) \, \frac{dx}{x} \\ \big[\mathcal{M}^{-1}\phi\big](s) &=& \int_{c-i \infty}^{c+i\infty}x^{-s} \, \phi(x) \, dx \end{eqnarray}
However, for the isometry we use a different Mellin transform $\mathcal{M}'$ $$ \big[\mathcal{M}'f\big](s) = \int_0^\infty x^{-\frac{1}{2}+is}\, f(x) \, dx = \big[ \mathcal{M}f \big](\frac{1}{2}+is) $$
and this will be an isometry between the two Hilbert spaces:
$$ \mathcal{M}': L^2(\mathbb{R}, dx) \to L^2(\mathbb{R}^\times , \frac{dx}{x}). \text{ with }|| f||_{L^2(\mathbb{R},dx)} = ||\mathcal{M}'f||_{L^2(\mathbb{R}^\times, \frac{dx}{x})} $$
So for example, there might be an analogy betwen Fourier series and Dirichlet series.
Since $$\int_0^\infty|f(t)|^2\frac{dt}t=\int_{-\infty}^\infty |f(e^x)|^2\,dx,$$ the mapping $Tf(x)=f(e^x)$ gives an isometry from $L^2((0,\infty),\frac{dt}t)$ onto $L^2(\Bbb R)$. You can use this isometry to obtain an orthonormal basis for either of those $L^2$ spaces from an orthonormal basis for the other.
But you should note that the supposed orthonormality relations you give are nonsense. The functions $t\mapsto n^{it}$ are not orthonormal in $L^2((0,\infty),dt/t)$. They can't be, since they're not even elements of that space! $$\int_0^\infty|n^{it}|^2\frac{dt}t=\int_0^\infty\frac{dt}t=\infty.$$For the same reason the functions $e^{inx}$ are not orthonormal in $L^2(\Bbb R)$. In both cases the integral in your supposed orthonormality relation does not exist.
So what is an orthonormal basis for $L^2(\Bbb R)$? If you define $\mu$ by $d\mu(x)=e^{-x^2/2}dx$ then the Hermite polynomials are an orthonormal basis for $L^2(\mu)$; you can get an orthonormal basis for $L^2(\Bbb R)$ from the obvious isometry.