I am interested in the following question: can we generalize Fourier theory to different $L^2(\mu)$ spaces, not just the Lebesgue measure? For example on $\mathbb{R}$, defining the Fourier transform of $f$ as $$ \mathcal{F}[f](\omega)=\int_\mathbb{R}f(x)e^{-i\omega x}d\mu(x). $$
Is there literature on this subject? Do any properties hold with this (in particular Plancherel’s theorem)? I have tried to look this idea up online but found nothing.
I am a beginner when it comes to Fourier theory, I apologize if it is a simple question.
Yes, in fact one can define it on measures on topological groups (besides integrable functions with respect to Haar measures). In any event, lets just focus on $\mathbb{R}^n$. There, the Fourier transform is well define on the space of complex (or finite real valued) Borel measures $\mathcal{M}(\mathbb{R}^n)$. There is also an inversion formula that shows that the Fourier transform a one to one function on $\mathcal{M}$. The Fourier transform in this setting is often used in probability theory under the name of characteristic function.
Definition: Let $\mu$ be a Complex measure on $(\mathbb{R}^d,\mathscr{B}(\mathbb{R}^d))$. The Fourier transform or characteristic function of $\mu$ is the function defined as $$\widehat{\mu}({\bf t})=\int\exp(i{\bf t}\cdot{\bf x})\mu(d{\bf x})$$
When $\mu\ll\lambda_d$ and $\mu=f\cdot\lambda_d$, then the Fourier transform of $\mu$ is closely related to the Fourier transform of $f$. and $\hat{\mu}(-2\pi i t)=\hat{f}(t)$.
Here are a few important results on uniqueness:
Theorem: Suppose that $\mu$ and $\nu$ are complex measures (measures of finite variation) on $\mathscr{B}(\mathbb{R}^d)$. Then, $\mu=\nu$ iff $\widehat{\mu}=\widehat{\nu}$.
Theorem: Let $\mu$ be a complex Borel measure on $\mathbb{R}$ and let $\widehat{\mu}$ be its characteristic function. Then
$$ \mu(\{a\})=\lim_{T\rightarrow\infty}\frac{1}{2T}\int^T_{-T} e^{-iat}\widehat{\mu}(t)\,\tag{2}\label{two} $$ 2. If $f\in\mathcal{L}_1(\mathbb{R},\lambda)$ then $$\begin{align} f(x)=\frac{1}{2\pi}\lim_{T\rightarrow\infty}\int^T_{-T}e^{-iyt}\widehat{f}(-t/2\pi)\,dt\tag{3}\label{inversion3} \qquad\text{a.s.} \end{align} $$ 3. If $\widehat{\mu}\in\mathcal{L}_1$, then $\mu\ll\lambda$ and $$\begin{align} \frac{d\mu}{d\lambda}(y)=\frac{1}{2\pi}\int e^{-ity}\widehat{\mu}(t)\,dt\qquad\text{a.s.}\tag{4}\label{density-prob} \end{align} $$
With regards smoothness:
Theorem: Let $\mu$ be a finite positive measure on $(\mathbb{R}^n,\mathscr{B}(\mathbb{R}^n))$. If $\partial^\alpha\widehat{\mu}(0)$ exits and is finite for all $|\alpha|=2m$ then, $\widehat{\mu}\in\mathcal{C}^{2m}(\mathbb{R}^n)$; furthermore, for all $\alpha\in\mathbb{Z}^n_+$ with $|\alpha|=2m$, $\int |x^\alpha|\,\mu(dx)<\infty$ and $\partial^\alpha\widehat{\mu}(t)=i^{|\alpha|}\int x^\alpha e^{ix\cdot t}\,\mu(dx)$.
Theorem: Suppose that $\mu$ is a complex measure on $(\mathbb{R}^n,\mathscr{B}(\mathbb{R}^n)$. If $$ \int_{\mathbb{R}^n}|x_j|^m|\mu|(dx)<\infty, $$ then the partial derivative $\partial^m_j\widehat{\mu}$ exists, is uniformly continuous, and $$\begin{align} \partial^k_j \widehat{\mu}(t)=i^k\int_{\mathbb{R}^n} x^k_j e^{i x\cdot t}\mu(dx),\quad 0\leq k\leq m. \end{align}\tag{1}\label{deriv-m-charac} $$ Moreover, if $|x|^m=\Big(\sum^n_{j=1}x^2_j\Big)^{\frac{m}{2}}\in\mathcal{L}_1(|\mu|)$, then $\widehat{\mu}\in\mathcal{C}^m(\mathbb{R}^n)$, and $$\begin{align} \widehat{\mu}(t)= \sum_{0\leq|\alpha|\leq m}\frac{i^{|\alpha|}}{\alpha!}t^\alpha \int x^\alpha\,\mu(dx) + o(|t|^m)\tag{2}\label{otaylor} \end{align}$$
Theorem: (Bochner--Herglotz) $\varphi:\mathbb{R}^d\rightarrow\mathbb{C}$ is the characteristic function of a finite nonnegative measure $\mu$ in $\mathscr{B}(\mathbb{R}^d)$ iff $\varphi$ is a bounded positive definite continuous function.