Intuition for how the Fourier transform "preserves information" and "changes basis"?

Question

I am an undergrad that has taken a few courses in real and complex analysis. I am trying to understand the Fourier transform better at a level of abstraction somewhere between "it moves from physical to frequency space" and a graduate level that talks about things like "Pointryagin dual spaces."

1) (Preserving information) My current intuition for integral transforms comes from measure-theoretic probability. My thinking from scratch: I know the probability distribution of a random variable is essentially the "measure assigned to all Borel sets," so it makes sense that knowing $E[f \circ X]$ for any measurable $f$ suffices to characterize the distribution of $X$. This means knowing $E[X^k]$ for all integers $k$ suffices by polynomial approximation, and so moments characterize a distribution. Then since the exponential function is defined as a power series, the object $E[\exp( tX)]$ tells us the moments, and thus the distribution, of the rv $X$, so a MGF characterizes a distribution. I later learned the MGF is sometimes called a "Laplace Transform," so it makes sense that the Laplace transform preserves information. I also know the Fourier transform is the "characteristic function" of a random variable, but I'm not sure what the analog of the above explanation is for the characteristic function, and in particular the need for complex numbers here and the intuition for what their resulting interpretation (in the above sense) puzzles me. Of course, if you think of the naïve idea that the Fourier transform goes from "physical to frequency space" then it's obvious that it preserves information. But I am looking for an explanation in the sense given above.

2) (Change of basis) I have read the Fourier transform is a "change of basis" and I have also heard the fact that the exponential function is an eigenfunction of the Laplace operator, but I'm not sure how these two operators are related. In particular, I know the Fourier transform sends $\mathcal{F}: L_2 \to L_2$, where that function space has a countable basis. So I would expect a "change of basis" to be something like $\hat{f} = \sum_i a_i \phi_i$ where $\phi_i$ are a basis set of functions for the space. But the Fourier transform involves an integral, not a sum, somehow implying an "uncountable infinite basis" if we view $\hat{f}(x) = \int f(t) \exp(-2\pi i x t) dt$ the values $f(t)$ as the co-efficients $a_i$ and the $\exp(-2\pi i x t)$ as the eigenfunctions. What's going on here? In what precise sense is it a change of basis?

Answer 1

Since you seem to understand the mathematical part, I'll give in for granted and just focus on the interpretation part. I'm not sure I understand your questions exactly, but I'll try to answer in a general sense.

I like to think of the Fourier transform as a sort of 'improper' change of basis. In fact, when we express a vector $\mathbf{v}$ given a set of basis vectors $\{ \mathbf{\phi}_i\}$ we write

\begin{equation} \mathbf{v} = \sum_j c_j \mathbf{\phi}_i \end{equation}

or, using coordinates:

\begin{equation} \label{eqn:v_e} v_i = \sum_k c_k \phi_{ik} \end{equation}

in which ${c_k}$ are the coefficients and $\phi_{ik}$ is a (really bad) notation for the $k$-th element of the $i$-th vector. This means that, given the basis, knowing the set of coefficients is the same as knowing the vector (which is in fact expressed in this basis by the coefficients themselves).

The same principle can hold for a function $f(x)$ that we can express as a combination of basis-function ${\phi_k(x)}$.

\begin{equation} f(x) = \sum_k c_k \phi_k(x) \end{equation}

and, just like before, for a given set of basis functions, knowing the coefficients gives us all the information we need to reconstruct the original function. This is the case, e.g., of the Fourier Series (a discrete version of the Fourier Transform).

Finally, if instead of using a discrete set of basis-function we use a continuous one we need to parametrize it using a continuous variable instead of discrete one ($\phi_k(x) \rightarrow \phi(k,x)$) . Moreover, the summation will turn into an integral ($\sum_k \rightarrow \int dk$).

\begin{equation} \label{eqn:f_e} f(x) = \int dk ~ c(k) \phi(k,x) \end{equation}

Note the similarity between this expression and the expression for $v$. In the Fourier transform, $\hat{p}$ are the coefficients, and the complex exponential functions are the basis on which we express $p$.

So, how does this relate to your questions?

1. Preserving information: Since by the knowledge of the Fourier coefficients we can can recover our original function, we have the same information about it. The F-transforms of a function is another representation of the function, but the function is still there. A good parallel might be representing a probability distribution using its PDF, using its CDF or is MGF: it is always the same distribution, just expressed in different terms.

2. Change of Basis Fourier series is a change of basis in the sense you are referring to. Fourier transform is an integral transform which can be seen as a sort of "change of basis" for the reasons I've explained above, but it is not as discussed for example in this reply.

Answer 2

Here are some notes I've taken in response to my questions by looking at various sources such as Terry Tao's blog, the Stakarchi book on Fourier analysis, and various answers here.

• Definition

  • $\mathcal{F}[f(x)] = \int_{-\infty}^\infty f(t)\exp(-2\pi i xt)dt$

• Various perspectives on the FT

  • Type

    • $\mathcal{F}: L_2(\mathbb{R}) \to L_2(\hat{\mathbb{R}})$, where $\hat{\mathbb{R}}$ is the "Pointryagin dual space" of the real numbers. The reason we often write $\mathcal{F}: L_2(\mathbb{R}) \to L_2(\mathbb{R})$ is because $L_2(\mathbb{R}) \simeq L_2(\hat{\mathbb{R}})$ are isomorphic, so it is easy to explain to undergrads as a "change of basis (of the same space)." In particular, the two spaces are different Hilbert spaces.
    • Note how $\hat{f}(x) = \int_{\mathbb{R}}f(t)\chi_x(t)dt$ can also be viewed as a composition $x \to \chi_x \to \hat{f}(x)$ from the isomorphism mentioned above. Thus, the FT outputs a function on the dual space (of functionals) because $\int_{\mathbb{R}}f(t)\chi_x(t)dt$ is a function that takes in $\chi_x \in \hat{\mathbb{R}}$ and outputs a scalar.
      • This abstract perspective generalizes the utility of this transform from merely physics and signal processing (when the LCA group in question is the reals) to general settings. We can do "frequency decomposition" in very abstract settings in a well-behaved way as a result. This is what harmonic analysis is all about!

  • In what sense is it a change of basis

    • The isomorphism is as $M: \mathbb{R} \to \hat{\mathbb{R}}$ given by $M: x \to \chi_y(x) = \exp(ixy)$. Thus as scalars are mapped to functions, functions on scalars $f(x)$ are mapped to functions on functionals, i.e., elements of $L_2(\hat{\mathbb{R}})$. In a way, the character functions $\chi_y(x)$ are analogous to the orthonormal basis $\phi_k(x)$ in an eigendecomposition in a finite-dimensional Hilbert space. It's just that there are uncountably many of them, so the sum in the eigendecomposition becomes an integral. In this sense, the FT is an "improper eigendecomposition." The "proper" decomposition in such a basis leads to Fourier series, only applicable to periodic functions.
      • This is a continuous group isomorphism.
    • The intuition is that each $x$ in characterizing $f(x)$ contributes a tiny amount to all the Fourier coefficients. Thus, taking into account the requirement for each $x \in \mathbb{R}$ in the domain, altogether, determines the Fourier coefficients.

  • The need for complex numbers and sinusoids

    • An important intuition is to do with circles. Consider paths on the plane. As we sweep $t$, we note that $Re^{i \omega t}$ draws a circle. Then, sweeping $t$ over $R_1 e^{i \omega t_1} + R_2 e^{i \omega t_2}$ allows us to hit a circle-within-a-circle motion. It turns out that (uncountably) infinitely many such circles allow us to reach any point in the plane. That is, any path $\gamma(t)$ on the plane can be written as $\gamma(t) = \int R_ke^{i \omega_k t}dk$. The key is that we need $\omega \in \mathbb{C}$ because we need to specify not just the speed of rotation on each circle but the starting point, to be able to express any such path in this way. Note that $e^{i \omega_k t}$ for $\omega_k \in \mathbb{C}$ is precisely a complex sinusoid!
      • If $\gamma(T) = \gamma(0)$ for some $T$, this reduces to a Fourier series. Thus we have Fourier series representations for any periodic function.

  • Why convolve with complex exponentials?

    • Fourier was motivated to solve differential equations like the heat equation. It was realized early that linear combinations of solutions to DEs were solutions, and that motivated the search for a function that could universally be used as a "basis" for solutions. The trigonometric functions and complex sinusoid (which is really the sum of sin and cosine) was a clear candidate because it is an eigenfunction of the differential operator. The choice of complex exponential vs. sin/cosine is not too important: Fourier himself defined his transform with the $\sin$ function. This is just a detail for some algebraic convenience.
      • This eigenfunction property is important because the FT tells us we can decompose any function into these complex sinusoids, so it suffices to study the behavior of various sinusoids under various differential operators because then we can reason about the behavior of any function under a differential operator.

  • Relation to MGFs and Characteristic Functions

    • The MGF is just the Laplace transform of the PDF. The Characteristic Function is then the Fourier Transform. Since these integral transformers are just "change of basis" in the way outlined above, it should be obvious that both an MGF and Characteristic Function characterize (duh) a distribution because a PDF characterizes a distribution.

  • Key properties

    • Linearity: $\mathcal{F}[a_1f_1 + a_2f_2] = a_1 \mathcal{F}[f_1] + a_2\mathcal{F}[f_2]$.
    • Rotation: $\mathcal{F}[f(x-a)] = \exp(-ika)\mathcal{F}[f(x)]$.
    • Convolution: $\mathcal{F}[f * g] = \mathcal{F}[f] \cdot \mathcal{F}[g]$.
    • Most usefully, differentiation: $\mathcal{F}[f'] = ik \mathcal{F}[f]$.
      • These four properties are very useful because they allow us to compute the FT of complicated functions using the FT of known simple functions.