I'm reading notes about M-estimators, and have within these notes been briefly introduced to manifolds, as a way to create what the author call "smooth hypothesis" for statistical models. A basic technical assumption of the statistical models we consider is that the set of possible true parameters of the model lies within $U \subset \mathbb{R}^d$ which is assumed to be open. We now define
Definition - Smooth hypothesis: Let $M \subset \mathbb{R^d}$ be a manifold in $\mathbb{R}^d$ of dimension $m$ (it's not stated whether $m\leq d$ or $m<d$). A smooth hypothesis is that the true parameter lies within $M\cap U$
The theory for M-estimators established in the notes rely heavily on the assumption that the parameter set is open, and therefore the author says that the theory of M-estimators can't directly be used on the model under the hypothesis.
Thus he indirectly says that $M\subset \mathbb{R}^d$ is not open. Why is this?
Also bare in mind that I have zero experience with advanced geometry.
He must have meant that $m<d$, because if $m=d$ then taking $M=U=\mathbb R^d$ is a counterexample. So assume $m<d$.
If $M$ is open in $\mathbb R^d$ then every point of $M$ contains a $d$-dimensional ball contained in $M$. The fact that $M$ is an $m$-dimensional manifold means that $M$ is locally homeomorphic to $\mathbb R^m$: every point of $M$ has a neighborhood that looks like an open subset of $\mathbb R^m$.
These two facts together imply that some open subset of $\mathbb R^m$ contains an open subset homeomorphic to a $d$-dimensional ball. By the invariance of domain theorem, this is impossible.
The invariance of domain theorem is one of those facts that seems intuitively obvious but is hard to prove. It's quite likely that your source assumes the theorem without proof.