Let $p$ be a prime number an $G$ be a group. If $|Z(G)|=p$ then $|G/Z(G)|=p$. This is a direct consequence of the first Isomorphism theorem, isn't it? Since $Z(G)$ is the kernel of the canonical projection.
Edit. I think my question breaks down to that I do not fully understand the first part of the last sentence in the proof below.
That's not true. Using the Isomorphism Theorem on the canonical projection of $G$ onto $G/Z(G)$ gives $|G|/|Z(G)| = |G/Z(G)|$ (domain order over kernel order = image order). In fact, |G/Z(G)| is never prime because G/Z(G) is never non-trivially cyclic but that's a different matter.