From time to time I come across the statement that the mathematical formalization of the mathematical instruments, which the quantum theory uses, is not complete yet (admittedly, more often in popular science texts). But unfortunately, my google/wikipedia-search wasn't really successful. So I wondered if that's actually true and if so, what some of those open questions actually are?
What I imagine, when I read those statements, are physicist 400 years ago, using unproven theorems in calculus until finally the Analysis was formalized in the 20th century. But to me, quantum theory seems to be making use only of basic functional analysis and probability theory, nothing really crazy.
Quantum mechanics is fine; when people say this they're talking about quantum field theory, which has not been put on a rigorous mathematical foundation. For example the Feynman path integral has not been made into rigorous mathematics; it requires that we discuss an integral with respect to a measure on an infinite-dimensional space of fields that has not been rigorously constructed. There are axiomatizations of quantum field theory available, such as the Haag-Kastler axioms, but my understanding is that no physically relevant examples of QFTs have been proven to satisfy any of the proposed axiomatizations.
One of the Millennium problems, Yang-Mills existence and mass gap, is about this.