I've proven that, given an equaliser diagram where the equaliser map $e : E \to X$ is an isomorphism, the equalised maps $f, g : X \to Y$ say, must be equal.
Conversely, if we assume the equalised maps $f$ and $g$ are already equal, I can find a (unique) right inverse $k : X \to E$ of $e$: $e \circ k = \text{id}_X$. First I thought that the uniqueness of $k$ would fix that this right inverse would also be a left-inverse of $e$, but I no longer think that to be true.
I now did realise that $X \xrightarrow{\text{id}_X} X\xrightarrow{f = g} Y$ is of course also an equaliser diagram. Thus, this yields a unique arrow $h : E \to X$ s.t. $\text{id}_X \circ h = e$. It seems to me the uniqueness and existence of these $h, k$ would now suffice to conclude that $e$ is an isomorphism. ($k = h^{-1} = e^{-1}.)$
But I didn't exactly find a left-inverse $h$ s.t. $h \circ e = \text{id}_E$, what I actually set out to do. So is this sufficient?
So you want to prove that if $f = g$ then the equaliser is an isomorphism. A quick proof would be to note that any two equalisers are isomorphic, so as the identity function $id_X$ is an equaliser we have that any equaliser is isomorphic to the identity, and is thus an isomorphism.
If you want to work this out in detail you essentially go through the proof that any two equalisers are isomorphic. I will use your notation. You have already found a right inverse $k$. Now note that $e \circ k \circ e = e = e \circ id_E$, so by the universal property of $e$ being an equaliser we have that $k \circ e = id_E$ and we see that $k$ is also a left inverse.