Say we have the floating-point system $(2,3,-1,2)$ and we want to find machine epsilon. According to my textbook, this can be found as $\epsilon_m=\beta^{1-t} = 2^{1-3}=0.25$. However, my textbook also says that $\epsilon_m$ represents the distance between number 1 and the nearest floating-point number such that $1 + \epsilon_m > 1$. Taking the closest numbers to 1 in this system gives me: $$(0.111)\times 2^0 = 0.875; (0.100)\times 2^1= 1 (0.101)\times 2^1 = 1.25$$
Then, since 0.875 is closer to 1 than is 1.25, why is $\epsilon_m$ not 0.125 instead of 0.25?
It's just a matter of definitions. By convention "machine epsilon" means the difference between $1$ and the next representable number, not the previous.
You quote the definition yourself -- you're looking for an $\epsilon$ such that $1+\epsilon > 1$. Since $0.875$ is not greater than $1$, you cannot use that as $1+\epsilon$. So $0.875$ is not the nearest representable number that is greater than $1$.