While reading chapter 4 of "Deep Learning Architectures, A Mathematical Approach" by Ovidiu Calin, I stumbled upon the following statement:
Using the fact that the Gaussian tends to Dirac measure, $\lim_{\sigma \to +0}G_\sigma(x) \to \delta(x)$, we have $$ \lim_{\sigma \to +0} f_\sigma(x) = \lim_{\sigma \to +0} (f*G_\sigma)(x) = (f*\delta)(x) = f(x), $$
where $f(x)$ is an integrable function on $\mathbb{R}$ and $G_\sigma(x) = \frac{1}{\sqrt{2\pi}\sigma}\exp(-\frac{x^2}{2\sigma^2})$ and $f_\sigma(x) = (f*G_\sigma)(x) = \int f(u)G_\sigma(x-u)du$.
From what I know, I wasn't very sure about this result. Unfortunately, the proof is omitted from the book so I tried to verify the claim by myself. It is not exactly the same but I was able to confirm the results below by following argument in Chapter 9 of "Real and Complex Analysis" by Walter Rudin.
Result 1: If $f$ is bounded and continuous at point $x$, then $$ \lim_{\sigma\to +0}f_\sigma(x) = f(x). $$
Result 2: If $1\leq p < \infty$ and $f\in L^p$, then $f_\sigma$ converges to $f$ in $L^p$ as $\sigma \to +0$.
From the second result, it follows that there exists a sequence $\sigma_n \to +0$ such that $f_{\sigma_n}(x) \to f(x)$ almost surely. But it does not extend to pointwise almost sure convergence of $f_\sigma \to f$.
It doesn't seem to be possible to show pointwise almost sure convergence $f_\sigma \to f$ without assumptions like in the result 1. But I'm not sure if there is something special about gaussian or if there is a counterexample to it.
I appreciate it if anyone could prove the claim or provide a counterexample to it. You don't have to write full solution, if you know a good reference, then just a reference is fine.
Update: As Jochen pointed out, by following the argument here, we can show almost sure pointwise convergence if $f \in L^\infty$. So, the question really is whether we can relax it to $f \in L^1$.
(Not exactly an answer, but too long for a comment)
If you only require $f$ to be integrable, the pointwise convergence fails on $\mathbb Q\cap[a,b]$ for arbitrary real numbers $a<b$.
Indeed, because we have $\int G_\sigma(x-u)du = 1$, we can write for all $\sigma>0$
$$ \begin{align*} f_\sigma(x) - f(x) &= \int_{\mathbb R} \left[f(u) - f(x)\right]G_\sigma(x-u)\ du\\ &:=\frac{1}{\sqrt{2\pi}\sigma} \int_{\mathbb R} \left[f(u) - f(x)\right]\exp\left(-\frac{(x-u)^2}{2\sigma^2}\right)\ du\\ &\stackrel{z\leftarrow (x-u)/\sigma}{=} \frac{-1}{\sqrt{2\pi}}\int_{\mathbb R} \left[f(x-\sigma z) - f(x)\right]\exp\left(-\frac{z^2}{2}\right)\ dz \end{align*}$$
So now let $f$ be the Dirichlet function on $[0,1]$, and $0$ everywhere else, i.e. $$f(x) := \begin{cases} 1 &\text{if } x\in[0,1]\setminus\mathbb Q\\ 0 &\text{otherwise}\end{cases} $$
It is clear that $f$ is integrable on $\mathbb R$, however for any $x\in\mathbb Q\cap(0,1)$, and any sequence of rationals $(\sigma_n)$ which converges to $0$, such as $\sigma_n \equiv \frac1n$, we have
$$\begin{align*}f_{\sigma_n}(x) - f(x) = f_{\sigma_n}(x) &= \frac{-1}{\sqrt{2\pi}}\int_{\mathbb R} f(x-\sigma_n z) \exp\left(-\frac{z^2}{2}\right)\ dz\\ &= \frac{-1}{\sqrt{2\pi}}\int_{\mathbb R\setminus\mathbb Q}f(x-\sigma_n z)\exp\left(-\frac{z^2}{2}\right)\ dz\\ &= \frac{-1}{\sqrt{2\pi}}\int_{(x-1)/\sigma_n}^{x/\sigma_n}\exp\left(-\frac{z^2}{2}\right)\ dz \\ &\stackrel{n\to\infty}{\rightarrow} -1\ne 0\end{align*}$$
Where we used the properties of the Lebesgue integral and the fact that if $z$ is irrational, then $x-\sigma z$ is irrational for any rational $x$ and rational $\sigma\ne 0$. By scaling and shifting, we thus see that it's possible to produce similar counter-examples for any subset of the form $\mathbb Q\cap[a,b]$.
This example shows that the author's claim can at best be true in the almost everywhere sense. I actually suspect that if we only require integrability, pointwise convergence also fails on positive measure subsets, but I haven't found such an example yet.