Naively, we might think that if a foundations of mathematics is consistent, then its fair game. Then we learn a bit more, and we realize that even if a foundations of mathematics is consistent, it may still prove false statements about the natural numbers. So, not every foundations of mathematics is 'trustworthy'.
Suppose that a foundations is not only consistent, but also, that it proves only true statements about the natural numbers. Does this exhaust the reasons a foundations might be 'untrustworthy', or are there further, more subtle reasons?
If a foundational system is based on axioms formulated in first-order logic, and it it is consistent, then it has a model, no matter how ridiculous its axioms might be. Your question therefore presupposes, quite reasonably, that we should not be satisfied with any only model but should demand models where at least the natural numbers of the model are the standard natural numbers, not some weird things where $0=1$ or where there is a Gödel number for a proof of a contradiction in ZFC.
It is possible for a theory to prove only true statements about the natural numbers and nevertheless not admit any models in which the natural numbers of the model are the standard natural numbers. You can produce such an axiom system as follows. Start with your favorite system that proves only true axioms about the natural numbers (perhaps Peano arithmetic, or perhaps ZFC). Enlarge its vocabulary by adding one new primitive constant symbol $c$; then add axioms saying "$c$ is a natural number", $c\neq 0$, $c\neq 1$, $c\neq 2$, etc. Clearly, any model of this axiomatic system must contain a non-standard natural number, namely the value assigned to the symbol $c$. Nevertheless, any statement $\theta$ about natural numbers (where "about" includes making sense in your original axiomatic system and thus not explicitly mentioning $c$) that is provable in this system is true. The reason is that a proof of $\theta$ would contain only finitely many of the new axioms, and all those axioms could be satisfied by giving $c$ a sufficiently large standard value. In fact, any statement that doesn't explicitly mention $c$ and is provable in the enlarged system was already provable in the original system.
In addition to wanting a foundational system to admit a model with the standard natural numbers, one might want this model to also have all the standard sets of natural numbers, real numbers, etc. Whether one actually wants this (and how far the "etc." should extend) is probably a philosophical issue, depending on whether you regard the "standard real numbers" or more complicated sets as well-defined concepts. I think most mathematicians regard "natural number" as a clear, unambiguous notion, but when it comes to sets of natural numbers, or sets of such sets, differences of opinion arise. For example, there are respectable mathematicians who do not consider the continuum hypothesis to be a clear statement. For someone with a thoroughly Platonist attitude, on the other hand, ZFC would be trustworthy but ZFC plus $V=L$ (Gödel's axiom of constructibility) might not be.