If a field requires every element that is not the additive inverse to have a multiplicative inverse, then how can a field have a cardinality that is not prime?
For instance 27?
So whether it is integers mod 27, or some direct product, there will be a nonzero zero divisor, hence not a unit. Does that not disqualify it from being a field?
Your argument shows that any field that is a quotient of the ring of integers must have a prime number of elements. Another way of saying this is that any finite field that has a single additive generator must have a prime number of elements, or more generally that the additive order of any individual nonzero element is prime.
However, not all finite fields are necessarily additively generated by a single element. The smallest example is $\Bbb Z_2[x]/(x^2+x+1)$, which has four elements. Here every element has order 2, and the additive group is the Klein Four group $\Bbb Z_2 \oplus \Bbb Z_2$. The multiplicative group of units is cyclic of order 3 in this case.