The incompleteness theorem states that one cannot prove whether ZF or ZFC is consistent, but what about ZFC withouth Axiom of infinity? (Assuming the empty set exists)
Furthermore, let $M$ be a consistent model not invoking infinity and $A,B$ be statements invoking infinity such that $A$ contradicts $B$. Then, let's assume both $M+A$ and $M+B$ are consistent. If statements $\phi_A$ and $\phi_B$ invoking infinity are provable in $M+A$ and $M+B$ relatively, then are finite pieces of $\phi_A$ and $\phi_B$ both provable in $M$?
The corollary from the incompleteness theorems is that you cannot prove the consistency of $\sf ZFC$ from $\sf ZFC$ itself. You have to have a stronger theory.
For example, in $\sf ZFC+\text{There exists an inaccessible cardinal}$, you can in fact prove the consistency of $\sf ZFC$ because this is a stronger theory.
Similarly this is the case of $\sf ZF_{fin}$ ($\sf ZF$ without infinity). The theory itself cannot prove its own consistency. However $\sf ZFC$ is a strictly stronger theory, and it proves the consistency of $\sf ZF_{fin}$. It does so by exhibiting a set which is a model of the theory, $V_\omega$ - the set of the hereditarily finite sets.
Large cardinal axioms are often called "strong infinity axioms" because they mimic the axiom of infinity, in the sense that they make a stronger theory by describing that a certain set of ordinals exists.