Say I have a complex Hilbert space, with the typical inner product:
$$ \langle u,v \rangle=u^\dagger v $$
The inner product for the zero element is also zero : $\langle 0,0 \rangle=0$
Say, I am to call the zero state an 'undesirable' state (it produces non-normalizable states for a wave-function, for instance). Can I remove it from the inner product? Can I define the inner product as follows:
$$ \langle u,v \rangle=\begin{cases} \nexists & \text{if } u=v=0 \\ u^\dagger v & \text{otherwise} \end{cases} $$
Quantum Mechanics seems to like it --- this is basically the inner product of allowed physical states. Is there any reason mathematicians don't like that? Is the space still a Hilbert space. If not what is it?
Of course you can define it this way. In mathematics, we may define anything whichever way we like. The question you should be asking is: does this definition make anything I'm doing easier?
We do not need ontological commitment to the entities of our models, and there's really no reason to try to "pare down" a model to only entities that seem "real," unless you can do it in a way that systematically avoids the need to manually handle edge-cases.
As far as I can tell, this does the opposite of that. It doesn't systematically change anything; all it does is introduce the need to propagate a bunch of extra conditions through all of the proofs. Anyone using the mathematics to actually describe a thing will be able to prune the nonphysical entities post-hoc - there's no need to tediously carry the caveats through each step of the math when they don't change the logical structure of the arguments.