Orthogonality of generalized eigenbases of self-adjoint operators

Question

As mentioned in another of my recent questions, given some operator on a Hilbert space, we can construct a Gelfand triple in order to "recover" eigenfunctions corresponding to elements of the spectrum which do not strictly have corresponding eigenfunctions in the Hilbert space "proper." For example, the momentum operator $i\frac{d}{dx}$ has no "proper" eigenfunctions in $L_2(\mathbb{R})$, but it does have sinusoidal eigenfunctions in the Gelfand triple $H^s(\mathbb{R}) \subseteq L_2(\mathbb{R}) \subseteq H^{-s}(\mathbb{R})$.

In the finite-dimensional case, we have a theorem that every Hermitian operator has an orthogonal eigenbasis. Does this generalize to the infinite-dimensional case of self-adjoint operators?

Immediately, we encounter a problem: the sinusoids $t \to e^{ikt}$ are not "proper" elements of $L_2(\mathbb{R})$ (they fail to be square-integrable). So, we cannot say precisely that these eigenfunctions are orthogonal with respect to the inner product on the Hilbert space. Can we state something weaker?

Perhaps we can recover a notion of "limiting orthogonality" by approximating our eigenfunctions with sequences of functions that are elements of $L_2(\mathbb{R})$ (e.g. approximating a sinusoid by a sequence of sinc functions with slower and slower decay)? Maybe there's some other approach entirely? Or is this a doomed endeavor, and there is no useful generalization to be found?

Answer 1

I don't intend to give a full answer (but annoying to write in the small comment box). I think that to "diagnolize" can be thought of as finding an isomorphism of your situation with the situation of the operator of multiplying by a function $\lambda (x)$ on a space $X$. So, for example, if you consider the operator $d/dt$ on functions on the real line, you can consider the operator of multiplying by the function $x \mapsto x$ on functions on a different copy of the real line, and we have an isomorphism between the situations, given by Fourier transform (so we have a "continuous" orthogonal basis, consisting of delta functions at different points of the second copy of the real line). So you need to look up spectral theorems for self-adjoint operators, and interpret them as giving isomorphism with $L^2$ of some measure space (the spectrum). This one interprets as finding an "orthonormal basis" (the delta functions at the various points of the spectrum, not really a basis of course, a formalization of a "continuous basis").

Answer 2

For bounded, self-adjoint operators $T$ on a Hilbert space, there is Weyl's notion of an "approximate eigenvector" for elements that are in the continuous spectrum (but not discrete). Namely, for $\lambda\in\mathbb C$, a sequence of norm-one vectors $v_n$ is an approximate eigenvector for $\lambda$ when $(T-\lambda)v_n\to 0$. Every non-eigenvalue $\lambda$ in the continuous spectrum has an approximate eigenvector, and conversely.