In a proof I read recently, I saw the author conclude that $X$ is an Exp(1) random variable after finding $E(X^{k}) = k!$. Why and when is this allowed (i.e. when can I conclude that some r.v. follows a distribution whose expectation I recognize)? Is it not possible to construct another random variable with expectation $k!$ that isn't an Exp(1) random variable (maybe I just missed something earlier on)?
I haven't been able to find much online about the injectivity of the expectation function, so maybe someone can clarify for me.
The claim is, that given $EX^k=k!$ for each $k=0,1,2,\dots$ implies $X$ has the indicated $\mathrm{Exp}(1)$ distribution. Most of the comments to your question ignore the statement that the $\mu_k=k!$ condition holds for all $k$, even though it is clear in the post you cite.
Read all about this kind of problem in general here and here. In your special case, since the series $$f(z)=\sum_{k\ge0} EX^k\frac {z^k}{k!}=\sum_{k\ge0}z^k = \frac 1{1-z}$$ converges for all complex $z$ in a neighborhood of $0$, there is only one probability distribution with the indicated moments.