It is well know that the operator $\exp\left[-i\Delta t\hat{H}\right]$ is a unitary operator if $\hat{H}$ is an Hermitian operator. The proof is based on Taylor expanding and affecting the dagger: $\left(\exp\left[-i\Delta t\hat{H}\right]\right)^{\dagger}=\left(\boldsymbol{\hat{1}}+\sum_{n=1}^{\infty}\frac{(-i\Delta t\hat{H})^{n}}{n!}\right)^{\dagger}=\boldsymbol{\hat{1}}+\sum_{n=1}^{\infty}\frac{(i\Delta t\hat{H})^{n}}{n!}=\left(\exp\left[i\Delta t\hat{H}\right]\right)^{-1}$.
However, in practical common problems of quantum physics, $\hat{H}$ denotes an operator which might become unbounded at some point. The most familiar one is the delta potential:
https://en.wikipedia.org/wiki/Delta_potential
where $\hat{H}=\delta(x)$. Clearly, for that choice there is no justification for Taylor expanding and I'm also unsure how to the conjugation is supposed to act on $\infty$.
Is it really true that the operator $\exp\left[-i\Delta t\hat{H}\right]$ is still unitary for the choice of $\hat{H}=\delta(x)$ ? I cannot even see how to even define such an object, but surprisingly, the current approach assumes that the answer is positive.