As far as I know, the Fourier transform is a (linear) unitary transform:
$T: \textbf{L}^2(-\infty, +\infty) \rightarrow \textbf{L}^2(-\infty, +\infty)$
where the basis functions {$e^{i \omega x} | \omega \in \textbf{R}, x \in \textbf{R}$} are not $L^2$-functions.
Question 1: does there exist such set of $\textbf{L}^2$ integrable basis in the $\textbf{L}^2(-\infty, +\infty)$? If it does exist, e.g. {$f(x) | x \in \textbf{R}$}, the Fourier transform of this set of basis must be a transform of basis in $\textbf{L}^2(-\infty, +\infty)$, since the $T$ is unitary - am I right?
(a little stretch on question 1: if there exist infinite number of sets of such basis, is there any general constructing/generating method to find one set of such basis?)
Question 2: the basis element $e^{i \omega x} \notin \textbf{L}^2(-\infty, +\infty)$, however, the Dirac $\delta$-function as follows:
$\delta(x - y) = \int_{-\infty}^{+\infty} { \frac{e^{i \omega x}}{\sqrt{2\pi}} \frac{e^{-i \omega y}}{\sqrt{2\pi}} d \omega}$
can be viewed as the Fourier (unitary) transform of the very own basis {$e^{i \omega x}$}. The Dirac functions {$\delta (x - y)$} can be viewed as the sort of "singular" basis (with the "singular" support). Interestingly, the $\delta (x - y)$ is $\textbf{L}^2$ integrable. Now the question about the uniqueness - is this the only possible "basis transformation" on the basis {$e^{i \omega x}$}? If not, what else can we have?
Given that I am not familiar with the relationship of the Fourier analysis and Hilbert spaces (yet willing to learn), what resources/books would you recommend, if I need to learn more?
If you have an integral transform $Uf = \int_{-\infty}^{\infty} P(s,x)f(x)dx$ with an inverse integral transform $Vg = \int_{-\infty}^{\infty} Q(x,s)g(s)ds$, then you'll have something similar to the Fourier transform $\delta$-function behavior because $$ \begin{align} f(y) & =(VUf)(y) \\ & =\lim_{R\rightarrow\infty}\int_{-R}^{R}Q(y,s)\int_{-\infty}^{\infty}P(s,x)f(x)dxds \\ & = \lim_{R\rightarrow\infty}\int_{-\infty}^{\infty}\left(\int_{-R}^{R}Q(y,s)P(s,x)ds\right)f(x)dx. \end{align} $$ So you can see how the kernel in last transform will behave as one would expect a $\delta$-type-function to behave as $R\rightarrow\infty$. Of course, making all of this rigorous is a mess. A couple of other examples of integral transform pairs are given in the table on this Wikipedia page: http://en.wikipedia.org/wiki/Integral_transform .
On the same Wikipedia page you'll also find examples of integral transforms on the half line; these are more common. Mostly such transforms come out of spectral theory for classic differential operators $$ Lf=\left(-\frac{d^{2}}{dx^{2}}+q(x)\right)f,\;\;\; 0 \le x < \infty, $$ with some conditions imposed at $0$ and possibly at $\infty$. In this context, a Fourier transform and an inverse Fourier transform will be built from the classical eigenfunctions of the operator $L$. That is, there is a spectral density measure $\mu$ such that $$ f=\int_{-\infty}^{\infty}\left(\int_{0}^{\infty}f(t)\phi_{\lambda}(t)dt\right)\phi_{\lambda}(x)d\mu(\lambda) $$ Forward Transform: The inner integral defines a unitary map $U : L^{2}[0,\infty)\rightarrow L^{2}_{\mu}(\sigma)$ where $\sigma$ is the spectrum of the operator $L$. The precise definition of this unitary map is $$ Uf = L^{2}_{\mu}(\sigma)-\lim_{r\uparrow\infty}\int_{0}^{r}f(t)\phi_{\lambda}(t)dt $$ Inverse Transform: The outer integral defines a unitary map $V : L^{2}_{\mu}(\sigma)\rightarrow L^{2}[0,\infty)$ by the $L^{2}[0,\infty)$ limit $$ Vg = L^{2}-\lim_{\Lambda\rightarrow\infty}\int_{-R}^{R}g(\lambda)\phi_{\lambda}(x)d\mu(\lambda). $$ These two transforms are truly inverses; they invert each other in either order. You can see in this context how one can interpret these operator inversions to mean $$ \int_{-R}^{R}\phi_{\lambda}(x)\phi_{\lambda}(t)d\mu(\lambda)\rightarrow \delta(x-t) $$ This ''wave packet'' phenomena of Physics is a general principle.
Your Case: These integral transforms look very much like continuous versions of the basis expansion $f=\sum_{\alpha}(f,e_{\alpha})e_{\alpha}$ for a Hilbert space. And, really, they basically are. For this case of "continuous spectrum" the individual eigenfunctions are not going to be square integrable. But when you sum using an integral over a bunch of small intervals, you can't really tell the difference. The case I've stated is general because it allows measures with discrete, singular, and continuous components. If the measure is purely discrete, then you get $$ f = \sum_{n=0}^{\infty}\left(\int_{0}^{\infty}f(t)\phi_{\lambda_{n}}(t)dt\right)\phi_{\lambda_{n}}(x). $$ If the measure is absolutely continuous with respect to Lebesgue measure $m$, then there is a positive spectral density function $d\mu = \kappa dm$ so that $$ f = \int_{-\infty}^{\infty}\left(\int_{0}^{\infty}f(t)\phi_{\lambda}(t)dt\right) \phi_{\lambda}(x)\kappa(\lambda)dm(\lambda). $$ And, in general, there can be a mix of these two, along with a possible singular measure, depending on how pathological the original potential $q$ may be.
You're interested in the case where the spectrum is purely absolutely continuous. Then everything reduces to that last case. Plus, you want to be able to extend to the full interval $L^{2}(-\infty,\infty)$ instead of $L^{2}(0,\infty)$ which is a little tricky. There are cases of purely continuous spectrum. Hamiltonian operators associated with periodic structures are such cases where the spectrum is banded and the second integral is then taken over bands.
As a final note, $\int_{a}^{b}\phi_{\lambda}(t)\phi_{\lambda}(x)\,d\mu(\lambda)$ is always in $L^{2}$ over a finite interval $[a,b]$. That's the case with the exponentials, too. For example, the spectral measure is uniformly $\frac{1}{2\pi}d\lambda$ for differentiation $$ \frac{1}{2\pi}\int_{-a}^{a}e^{i\lambda x}e^{-i\lambda t}d\lambda = \frac{\sin(a(x-t))}{\pi(x-t)} $$ This function is in $L^{2}$. The same thing happens for the above case. In the case of only absolutely continuous spectrum, $\phi_{\lambda}(t)$ will never be in $L^{2}$, but, as I mentioned, integrals over finite intervals always are in $L^{2}$.
History: All of these types of integral transforms arise as integrals of eigenfunctions. That's how Fourier defined the original transform, too. That's where all of this started. And it's the origin of general spectral theory. A lot went into trying to understand the things you're asking about, and it's still going on.
References:
[1] does a nice job of introducing orthogonal function expansions, and solving boundary value problems. This is old, but maintained in current print for 50 years or more; you'll learn a lot generally and specifically. [2] has good information about transform methods. [3] is an original source from the man who offered many of the first rigorous pointwise convergence proofs for integral expansions and discrete expansions; the first chapters are excellent, deep and compact. Titchmarsh deliberately avoids too much Hilbert space theory in favor of pointwise results. [4] has a good condensed treatment of much of Titchmarsh's early chapters; this requires advanced-calculus level skills. [5] offers an introductory treatment of Functional Analysis that is a verbose, well-written approach that doesn't require too much graduate level material. Contains three different readable proofs of the Spectral Theorem. [5] is a great graduate-level math book from a pioneer in linear and non-linear analysis.