I apologize if this is a stupid question.
I'm under the impression that we cannot devise a set of axioms that results in a provably consistent mathematical system. Why is that?
If my impression is wrong, what exactly about ZFC causes it to have unprovable consistency?
Edit: I'd like to ask an additional question. If we started with the "simplest" axiom from ZFC, and started adding more and more axioms from ZFC, at what point would the consistency become unprovable?
Whether we can prove the consistency of some given system $X$ or not depends on what mathematical system we are working in (often referred to as "the metatheory"). If we use a strong metatheory we can prove more things than a weaker one, including proving that more systems are consistent, which is nice. On the other hand, if we use too strong of a metatheory, then we can prove wrong things, or even prove everything, and that's no good.
Strong systems can often prove the consistency of weaker ones. For instance, MK proves the consistency of ZFC, which proves the consistency of Z, which proves the consistency PA, which proves the consistency of PRA. But Gödel's theorem says roughly that no system (worth a hill of beans) can prove its own consistency, so a weaker system can prove a stronger system consistent.
So if we take our metatheory to be ZFC, then there you go... Gödel tells us that the consistency of ZFC is unprovable there. But the consistency of lots of weaker systems, e.g. Z, PA, PRA is provable.
To your last question, if again we're taking ZFC as our metatheory, it happens to be the case that ZFC proves the consistency of any finite subtheory of ZFC, so you can keep adding axioms forever in that sense. If you're wondering how many we can delete before it becomes provably consistent, the answer is 1 (if you choose the right one)... e.g. ZFC-P (where P is the power set axiom) is provably consistent in ZFC.