In the topos $MSet$, the subobject classifier $\Omega$ is the set of all left ideals of $M$.Why each element of $\Omega^X$ can be identified with a family $(X_m)_{m\in M}$ with $nX_m\subseteq X_{nm}$?
And why actions are given by $n(X_m)_{m\in M}=(X_{mn})_{m\in M}$ for $n \in M$?
And finally, why $M$ is a group if and only if $\Omega=\{\emptyset,M\}$?
Let's dispose of the easiest part first: a group cannot have any nonempty left ideals beside the whole group. Conversely, if $\emptyset$ and $M$ are the only left ideals of the monoid $M$, then any $a \in M$ has a left inverse because the left ideal $Ma$ contains $1$; the left inverse $a'$ of $a$ also has a left inverse because the left ideal $Ma'$ contains $1$. It follows that every element of $M$ is invertible.
Now for $MSet$. I will use left actions of $M$ and will write morphisms of $M$-Sets on the right side of their argument, so that the compatibility condition for a map $\varphi:\, X\to Y$ conveniently looks like $(mx)\varphi = m(x\varphi)$. Every $n\in M$ also gives a morphism from $M$ to $M$ by right-multiplication.
(1) The basic observation is the special case of the Yoneda Lemma, that for every $MSet$ $X$, there is an isomorphism:
$$ MSet(M,X) \ni \varphi \leftrightarrow (1\varphi) \in X $$
This is in fact an isomorphism of $M$-Sets where the action of $M$ on $MSet(M,X)$ is given by precomposing with right-multiplication, i.e. for $m\in M$ and $\varphi: M \to X$ the map $m\varphi$ is defined as $n(m\varphi) = (nm)\varphi$.
(2) In particular there are isomorphisms
$$ \Omega^X \cong MSet(M,\Omega^X) \cong MSet(X\times M,\Omega) $$
where the second isomorphism comes from the cartesian closedness and where $m\in M$ acts on $MSet(X\times M,\Omega)$ by precomposing with $id_X\times m$ (this ensures that the second isomorphism is indeed a morphism of $M$-Sets).
(3) from (2) we have that elements of $\Omega^X$ correspond to subobjects ( = $M$-invariant subsets ) of $X\times M$. For $S\subseteq X\times M$ to be invariant is just the condition:
$$\forall (x,m) \in S:\, \forall n \in M:\, (nx,nm) \in S$$
Now let $X_m := \{ x \in X \mid (x,m) \in S \}$. Then the above condition translates into
$$\forall m\in M:\, \forall x \in X_m:\, \forall n \in M:\, nx \in X_{nm}$$
In fact I think that $X_m$ is very poor notation because it hides the subobject from which that family is made of; $S_m$ would be better.
(4) The action of $M$ on $\Omega^X$ is given on $MSet(M,\Omega^X)$ as in (1) and transported to $MSet(X\times M, \Omega)$ as follows: for a subobject $S\subseteq X\times M$ and $n\in M$ we have
$$ (x,m) \in nS \iff (x,mn) \in S $$
which translates into your description of the action. So in clearer notation $(nS)_m = S_{mn}$.