Let $f: U \subseteq \mathbb R^n \to \mathbb R^m$ be a smooth map. We say $y \in \mathbb R^m$ is a regular value of $f$ if and only if all points in the set $f^{-1}(y)$ are regular. (see e.g. the definition 4.12 given here).
A critical value on the other hand, is defined to be any $y=f(x)$ such that $x$ is a critical point.
Why is a critical point not defined to be any point $y$ such that all points in $f^{-1}(y)$ are critical?
It's very simple. You want to get all points, not some. Regular values are well-behaved and then there's everything else. By the way, it would be a rare occasion indeed for every point with the same value to be a critical point. (Just draw graphs of functions $\Bbb R\to\Bbb R$.)