In Quantum Mechanics one often deals with wavefunctions of particles. In that case, it is natural to consider as the space of states the space $L^2(\mathbb{R}^3)$. On the other hand, on the book I'm reading, there's a construction which it's quite elegant and general, however it is not rigorous. For those interested in seeing the book, it's "Quantum Mechanics" by Cohen-Tannoudji.
The book proceeds as follows: the first postulate of Quantum Mechanics states that for every quantum system there is one Hilbert space $\mathcal{H}$ whose elements describe the possible states of the system. The idea then is that $\mathcal{H}$ doesn't necessarily is a space of functions.
Indeed, Cohen defines (or doesn't define) $\mathcal{H}$ as the space of kets $|\psi\rangle\in \mathcal{H}$, being the kets just vectors encoding the states of the system.
The second postulate states that for each physically observable quantity there is associated one hermitian operator $A$ such that the only possible values to be measured are the eigenvalues of $A$ and such that
If $A$ has discrete spectrum $\{|\psi_n\rangle : n \in \mathbb{N}\}$ then the probability of measuring the eigenvalue $a_n$ on the state $|\psi\rangle$ is $\langle \psi_n | \psi\rangle$ considering that $|\psi\rangle$ is normalized.
If $A$ has continuous spectrum $\{|\psi_{\lambda}\rangle : \lambda \in \Lambda\}$ then the probability density on state $|\psi\rangle$ for the possible eigenvalues is $\lambda \mapsto \langle \psi_\lambda | \psi\rangle$
If, for example, the position operator $X$ for particle in one-dimension, exists, and if its eigenvectors are $|x\rangle$ with eigenvalues $x$, for each $x\in \mathbb{R}$, the probability density of position is $\langle x |\psi\rangle$ which is a function $\mathbb{R}\to \mathbb{C}$ and we recover the wavefunction.
This formulation, though, seems to be more general. In that case, wavefunction is just the information about one possible kind of measurement which we can obtain from the postulates. There is nothing special with it.
Now, although quite elegant and simple, this is not even a little rigorous. For example: the position operator hasn't been defined! It is just "the operator associated to position with continuous spectrum", but this doesn't define the operator. On the book, it is defined on the basis $\{|x\rangle\}$, but this set is defined in terms of it, so we get circular.
Another problem is that usually we are dealing with unbounded operators which are not defined on the whole of $\mathcal{H}$. And an even greater problem is that $\mathcal{H}$ was never defined!
I've been looking forward to find out how to make this rigorous, but couldn't find anything useful. Many people simply say that the right way is to consider always $L^2(\mathbb{R}^3)$, so that all of this talk is nonsense. But I disagree, I find it quite natural to consider this generalized version.
The only thing I've found was the idea of rigged Hilbert spaces, known also as Gel'fand triple. I've found not much material about it, but anyway, I didn't understand how it can be used to make this rigorous.
In that case, how does one make this idea of space of states, or space of kets, fully rigorous, overcoming the problems I found out, and possibly any others that may exist? Is it through the Gel'fand triple? If so, how is it done?
First remark. QM postulates state that the Hilbert space is separable. Recall that Riesz-Fischer theorem ensures that separable Hilber spaces are all isometrically isomorphic, so it does not really matter which one you choose until you are speaking about the general theory. In concrete realizations, it is obviously useful pick the Hilbert that best reflects the properties of the system.
Second remark. QM postulates don't and can't say how operators are made in specific situations. This is an experimental fact. They only say that these must be linear operators (generally unbounded) on $\mathcal H$, self-adjoint (not simply Hermitean!) if representing observables (i.e., quantities that can be actually measured on the system). It is a consequence of the noncommutative structure of the observables $C^*$-algebra that there exist couples of incompatible observables. (This leads to the Heisenberg principle.) Who says how operators are made? This is a prerogative of the quantization procedure. Such a procedure establishes a correspondence between classical and quantum observables, making precise the intuitive idea by Dirac. In a precise sense (Groenewold theorem), does not exists a "universal" quantization procedure (again, observables are experimental). One requires that $X$ and $P$ must be implemented one as a multiplication operator and the other as a differential operator (which one is a matter of taste, leading to the so-called x-representation and p-representation), continuous and essentially self-adjoint when considered on rapidly decreasing functions, self-adjoint (not both bounded) when considered on a maximal domain of $L^2$, satisfaying canonical commutation rules. [For quantities that have not a classical counterpart, such as spin, the actual form is induced extrapolating by experimental data algebraic properties such as commutation rules, spectra and so on.]
Third remark. Wave functions are a very peculiar kind of state. $\mathcal H$ has a lot of more other elements. Precisely, let $\mathcal A$ be the $C^*$-algebra of the physical system. Gleason theorem establishes a 1-1 correspondence between trace-class operators in $\mathcal A$ and the rays of a projective Hilbert space $\mathcal H$. The so-called wave functions are associated to the projectors of the form $(\psi, \, ) \psi$ (round brackets stands for scalar product in $\mathcal H$, $\psi \in \mathcal H$.). Physicists refer to $\psi$ as the wave function and this is misleading and, strictly speaking, uncorrect.
Gel'fand triple and Gel'fand-Naimark-Segal (GNS) construction constitute a way of automatically get the correct Hilbert space for the system. However, it is not something you can hope to find in a QM textbook for physicists, since the mathematical apparatus of QM is so strong that usually one can forget technical details, "canonically" realize the Hilbert space and the observables and perform actual calculations, which are the only important things in physics.
If you are interested in the last part, you can see
[1] Bogolioubov (et alii), Axiomatic quantum field theory
[2] Lansdmann, Mathematical topics between classical and quantum mechanics
[3] Dixmier, $C^*$-algebras (without applications to QM)