My question is very vague and poorly stated. Let $f: M \to \mathbb{R}$ be a smooth function with $M$ a smooth manifold. Let $y \in \mathbb{R}$ be a critical value which isn't a global extremum. We know in general that $f^{-1}(y)$ isn't necessarily a submanifold of $M$, but I'm wondering if there are cases where it still is, or at least is "almost" a manifold.
To be more clear about what I'm saying I'll give an example that's easy to visualize. Say $M$ is a compact surface in $\mathbb{R}^3$ and $f$ is the height function. For a regular value $x$, the preimage of it will look like a set of circles (if I'm not wrong). I think of it as the intersection of the surface with the plane $z=x$
For a critical value $y$ that isn't a global max or min (say it is the peak of some hill), the preimage will consist of a point as well as at least one circle. This is because $f^{-1}(y)$ not only has a critical point, but also a set of regular points which map to the circle, let's say.
How do we describe this preimage, is it still a manifold? It has components of different dimensions, does this contradict the definition? Can we say that removing the critical points of the preimage leaves us with a properly defined manifold?
There is a theorem in Lee's Introduction to smooth manifolds which states that for all closed subset $K$ of a smooth manifold $M$, there is a smooth function $f:M\to\mathbb{R}$ such that $f^{-1}(0)=K$ (!). So there is no hope in the general case to put structure on the preimage of any critical value.
But you can ask your function to be Morse, that is every critical point is non-degenerate, i.e. you can find coordinates $\varphi:U\to\mathbb{R}^n,p\mapsto (x_1(p),\dots,x_n(p))$ centered on each critical point $c\in U$ such that in these coordinates,
$$f(x_1,\dots,x_n)=f(c)+x_1^2+\dots+x_k^2-x_{k+1}^2-\dots-x_n^2.$$
This tells you that the preimage of a critical value of a Morse function is locally homeomorphic either to an euclidean space (near regular points of the sublevel) or to a $0$-level set of a quadric (near critical points of the sublevel).