The Greeks were initially convinced that all numbers were rational until upon pain of contradiction were forced to accept that $\sqrt{2}$ was irrational and needed to be included in our number system for geometric reasons. All other square roots are also necessary for the same reason, and so is $\pi$. $e$ is useful for other purposes but not really necessary in order to satisfy geometric needs. So you can define and use $e$ but you really need $\sqrt{2}$ because we are committed to certain other facts in Geometry.
So my question is, if we wanted a minimal number system that could handle our needs in applied mathematics (e.g., Physics and Geometry), would we be forced to accept any numbers besides the rationals, square roots, and $\pi$?
I know that by lacking the other real numbers you have an incomplete topological space, notions of convergent sequences become harder, and so on, but that's all for mathematical ease, not for theoretical necessity. Or am I wrong about that?
Nice question! Admittedly, the elusive real numbers that are neither computable nor definable are perhaps of limited use in applications in physics and elsewhere (see links provided in the comments).
As you point out, formal aspects such as existence of least upper bound are harder to handle if one works with a subset of the reals.
One further extension of your question is, if we already extended the ordered number system from the naturals to the integers to the rationals to the reals, why not throw in the familiar infinitesimals to form an ordered system where analysis can be done the way Leibniz, Euler, and Cauchy did it? Such a broader number system is useful in applications, such as modeling the phenomenon of small oscillations, by giving precise meaning to the idea that the period is independent of the amplitude without engaging in paraphrases via limits; see this article in Quantum Studies: Mathematics and Foundations.