The kernel of a monoid homomorphism $f : M \to M'$ is the submonoid $\{m \in M : f(m)=1\}$. (This should not be confused with the kernel pair, which is often also named the kernel.)
Question. Which submonoids $N$ of a given monoid $M$ arise as the kernel of a monoid homomorphism? (If necessary, let us assume that $M$ is commutative.)
Here is a necessary condition: If $xy \in N$, then $x \in N \Leftrightarrow y \in N$.
Let $(M,\cdot,1)$ be a monoid. A submonoid $N$ of $M$ is a kernel of some homorphism if and only if $y_1,..,y_n \in N$, then $x_1y_1...x_ny_n\in N \Leftrightarrow x_1...x_n \in N$. One implication is trivial. For the other we need to generate a congruence from $N$. Let $R$ be the transitive closure of the following set $$S=\{(a,b)\,|\,\exists a_1,...,a_n,b_1,...,b_m \in M, x_1,...,x_n,y_1,...,y_m\in N\ x_1a_1...x_na_n=y_1b_1...y_mb_m,\, a=a_1...a_n,\, b=b_1...b_m\}$$ which is submonoid of $M\times M$ and is reflexive and symmetric. Note that $R$ is hence a congruence. Let $q:M\to M/R$ be the canonical quotient. We will show that $N$ is the kernel of $q$. i.e. we need to show that if $(a,1)$ is $R$, then $a$ is in $N$. Let us begin by showing that if $(a,b)$ is in $S$ and $b$ is in $N$, then $a$ is in $N$. Accordingly suppose there exists $a_1,...,a_n,b_1,...,b_m\in M$, $x_1,...,x_n, y_1,...,y_m\in N$ such that $a=a_1...a_n$, $b=b_1...b_m$ and $x_1a_1...x_na_n=y_1b_1...y_mb_m$. We have $$b=b_1...b_m\in N \Rightarrow x_1a_1...x_na_n=y_1b_1...y_mb_m \in N \Rightarrow a=a_1...a_n \in N.$$ Now suppose that $(a,1)$ is in $R$, then there exists $u_1,...,u_p$ such that $u_1=a$, $u_p=1$ and $(u_i,u_{i+1}) \in S$. We have $(u_{p-1},u_p) \in S$ and $u_p=1$ is in $N$ which implies that $u_{p-1}$ is in $N$. Repeating we find that $a$ is in $N$ as required.
Perhaps it is worth mentioning that when $M$ is commutative the condition above reduces to the one mentioned in the question, and given $N$ the congruence $R$ above is simply $$R=\{(a,b)\,|\,\exists x,y\in N\, xa=yb\}.$$