When I was first introduced to Fourier transform, its core was a formula for it, something like:
$$\tilde f(k)=\int_{-\infty}^{\infty} e^{-2\pi i kx}f(x)\text{d}x.\tag1$$
It works nice for good enough functions, for which it converges, for example $\text{sinc}(x)$. But Fourier transform is also used for some more exotic functions, for which $(1)$ diverges, for example $x^2$, or even $\text{sgn}(x)$.
Some sources find Fourier transforms of such functions by regularizations (e.g. multiplying integrand by $\exp(-ax^2)$ and then taking the limit $a\to0$). But this looks like a dirty hack to me.
Obviously, there has to be some more general Fourier theory, than the one involving $(1)$, to handle such functions. What is such a theory? What books present this theory in an accessible way?
You're looking for tempered distributions, see for example https://www.math.ucdavis.edu/~hunter/book/ch11.pdf
Briefly, tempered distributions are distributions acting on Schwarz functions. Using the fact that the usual Fourier transform maps the Schwarz space into itself, this can be used to define the Fourier transform of a tempered distribution (via Plancherel's formula) as
$$ \langle \hat u, \phi \rangle = \langle u, \hat \phi \rangle $$
(other normalizations are possible, and common).