Given a finite semigroup $S$ containing a unique idempotent $e$, I can show that every element $s\in S$ has an ''inverse" $s^{-1}\in S$ in the sense that $s s^{-1} = s^{-1}s = e$:
Since $S$ is finite, every element $s$ has an idempotent power $s^{p_s}$ with $p_s\geq1$. It follows that $s^{2p_s}$ is also idempotent. Since $e$ is the only idempotent, we obtain $e=s^{2p_s}$. Since $2p_s\geq 2$, we can define $s^{-1} = s^{2p_s-1}$ in the sense stated above.
Now my question: Is $e$ actually a unit of $S$, i.e., does $es = se = s$ hold for every $s\in S$?
What you're trying to prove is not true. Consider for example the two-element semigroup $\{e, a\}$ with the operation $\ast$ that always outputs $e$ ("the null semigroup"). It's clearly associative, and $e$ is the unique idempotent, but this is not a group.