Suppose for the sake of argument that we look at ZFC with the axiom of infinity removed.
http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems#First_incompleteness_theorem
http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems#Second_incompleteness_theorem
We would then be in a position where the hypotheses of Gödel's theorems are not satisfied, correct? Basically, I want to remove, for the sake of argument, a minimal amount of axioms of ZFC so that ZFC minus some axiom(s) leads to a theory that does not include arithmetical truths and is not capable of expressing elementary arithmetic.
Is it possible that ZFC- (my shorthand for ZFC with some axiom(s) removed) is consistent and complete?
ZF minus the axiom of infinity plus a new axiom negating the axiom of infinity is bi-interpretable with Peano Arithmetic. And the two equivalent theories are both subject to Gödelian incompleteness.
For the equivalence claim, see Richard Kaye and Tin Lok Wong's paper On interpretations of arithmetic and set theory (which incidentally explores some of the complications alluded to in the answer by @zyx).