Suppose my language is $\mathcal{L} = \{+,-,\cdot,0,1\}$. My attempt is as follows:
$$ \forall x \forall y \forall z (x - y = z \leftrightarrow x = y + z)\\ \forall x x \cdot 0 = 0\\ \forall x \forall y \forall z (x \cdot (y \cdot z) = (x \cdot y) \cdot z)\\ \forall x x \cdot 1 = 1 \cdot x = x\\ \forall x \forall y \forall z (x \cdot (y + z) = (x \cdot y) + (x \cdot z))\\ \forall x \forall y \forall z ((x + y) \cdot z = (x \cdot z) + (y \cdot z))\\ \forall x \forall y x \cdot y = y \cdot x\\ \forall x (x \ne 0 \rightarrow \exists y ~ x \cdot y = 1)\\ \forall x \exists p ~~p \cdot x = 0 $$
My primary problem is with the last axiom, I do not know how to express the fact that $p$ should be a prime.
We cannot. Suppose that a certain first-order theory $T$ has every finite field as a model. Then $T$ has arbitrarily large (finite) models, and therefore by the Lowenheim-Skolem Theorem it has infinite models of every cardinality.
Alternately, we can use the Compactness Theorem to show $T$ has infinite models. It follows that there is no first-order theory whose models are precisely the finite fields.
Remark: It seems to me that in the OP you were trying to formalize the notion of non-zero characteristic. Using a standard compactness argument, one can also show that there is no first-order theory $T$ whose models are precisely the fields of non-zero characteristic.