I am having a hard time understanding why $\langle x|x^{\prime}\rangle=\delta(x-x^{\prime})$. Is this a purely formal expression as a consequence of $|\psi\rangle=\int|x\rangle\langle x|\psi\rangle dx$?
I think some abuse of notation is happening and we kind of extend the inner product to some space containing the Hilbert space $\mathcal{H}$ where the inner product is originally defined.
To get $\langle x | \psi \rangle = \int \langle x | x' \rangle \langle x' | \psi \rangle \, dx$ where $\langle x | \phi \rangle = \psi(x)$ we need to have $\langle x | x' \rangle = \delta(x-x')$.
I think that the notation is beautiful, but it is be a bit difficult to translate everything to rigorous mathematics.