It is true in general that the set of all invertible elements of a Monoid form a subgroup. The proof is trivial.
However, after some thought, I feel that if we restrict invertible to left or right invertible only, then it does not form a group. It seems so because I cannot imagine a way to prove otherwise.
I am looking for examples of Monoids whose left invertible elements do not form a subgroup or else proof that it does form a group.
On
Consider the bicyclic monoid. It is the quotient of the free monoid on two letters $a$ and $b$ by the relation $ab = 1$. The elements of this monoid can be written as $b^ma^n$ for some $m, n \geqslant 0$. The product of such words is given by $(b^ma^n)(b^pa^q) = b^ra^s$ where \begin{align} r &= m - n + \max(n,p) = m + p - \min(n,p)\\ s &= q - p + \max(n,p) = n + q - \min(n,p) %\label{bicyclic} \end{align} In particular $a^nb^n = 1$ and hence every element of the form $a^n$ has a right inverse and every element of the form $b^n$ has a left inverse. However, the only inversible element is $1$.
You can also describe this monoid as $\mathbb{N} \times \mathbb{N}$ with the product given by $$ (m, n) (p, q) = (r, s) $$ where $r$ and $s$ given as above.
Take the space $V$ of polynomials in $x$ over the reals (any other field would also work). Let us look at the monoid $End(V)$.
Derivation w.r.t. $x$, call it $D$, is an endomorphism of $V$. It has a right inverse gotten by integration. More precisely, if $P(x)$ is a polynomial, let's define $$ I(P):x\mapsto\int_0^xP(t)\,dt. $$
But because $D$ is not injective (as $D1=0$) it cannot have a left inverse. Thus the right invertible elements of $End(V)$ don't form a group.
Similarly we see that $I$ has no right inverse, because it is not surjective (the constant polynomial is not in its image). So the left invertible elements don't form a group either.