The following are a couple excerpts of the first chapter of Sakurai and Napolitano, Modern Quantum Mechanics, 2nd edition:


Prior to these formulas, the text discusses the fundamental mathematics of quantum mechanics with finite dimensional state spaces, in particular spin $\frac{1}{2}$ systems. The left hand sides of the formulas above are associated with cases involving finite dimensional (or at least countable) state spaces and the right hand sides are corresponding equations for continuous spectra.
I understand everything on the left hand side. In particular:
- $|\alpha\rangle$ is a vector in a separable Hilbert space.
- $\langle \cdot | \cdot \rangle$ is the Hilbert space inner product
- $\langle \alpha | \beta \rangle \in \mathbb{C}$.
But I'm confused by the formulas on the right hand side. What are the types of objects involved? For example, should one still think of $\langle \cdot | \cdot \rangle$ as a Hilbert space inner product yielding a complex number? If so, how can one interpret the right hand side of (1.6.2a) without some high intensity hand waving? The $\delta(\xi'-\xi'')$ expression suggests one should think of $\langle \xi'|\xi''\rangle$ as some type of linear operator, not an ordinary complex number.
I am also tempted to make the integrals on the right hand side disappear by thinking of something (maybe $|\xi'\rangle$?) as an integral operator as studied in functional analysis. Composing or applying integral operators may then yield integral expressions, but the linear operator perspective would be more fundamental and enlightening.
Any help from domain experts would be greatly appreciated.
I'm a physicist and not a mathematician so I can't answer all your questions, but perhaps I can help a bit.
As far as I know state vectors $\mid \psi \rangle$ are always the elements of a separable Hilbert space though there can often be additional mathematical structure.
The vectors $\mid \xi \rangle $ are parametrized by a real valued variable $\xi$. This isn't really all that different from the finite case where the basis vectors are parameterized by a discrete integer variable. In both cases the inner product between two basis vectors is a complex valued function in two variables,
$$\langle \xi \mid \xi' \rangle = f(\xi,\xi'),$$
As you noted above the dirac delta function is used all over the place in the formal theory quantum mechanics. This is not really a function, but rather a generalized function which is an equivalence class of ordinary functions. This is similar to the way in which real numbers are equivalence classes of sequences of rational numbers.
Before this theory of generalized functions existed Von Neumann came up with the theory of rigged Hilbert Spaces as the underlying mathematical structure of quantum mechanics. I don't personally know much about this other than the fact that the main goal of the theory was to avoid the use of the delta function. His book on quantum mechanics is probably going to be one of the most mathematically rigorous treatments of the theory that you will find.
One example of an inner product which results in an ordinary complex number is the inner product between the position eigen-states and the momentum eigen-states.
$$ \langle x \mid p \rangle = \frac{1}{\sqrt{2\pi \hbar} } e^{ipx/\hbar} $$
As far as interpreting the integrals and integral operators that should be fine. I do think that getting used to the notation gives a greater insight to the theory though.