This question seems to me to be the foundation of the concept of a mathematical proof. If we have proved a statement we automatically conclude that the statement is correct. But why can we do this? Is there some sort of "proof" that this works? Or do we need to assume that our set of axioms are not contradictory?
Since I haven't learned any mathematical logic, I don't know how to think about questions like this. Furthermore it seems to me that this question can only be answered rigorously if first is defined what a proof actually is and then showed that if we have a proof of a statement – satisfying the definition of a proof – that then the statement is true.
We need to assume more than just that the axioms are not contradictory. It is easy to see that if we prove a theorem in a way that satisfies the following two assumptions, the theorem must be true:
Each of the axioms we use in the proof is true
Each of the inference rules we use in the proof produces (only) true statements when applied to true statements
The challenge is how to justify (1) and (2). If we try to "prove" formally that our axioms are true, we will have to prove that the axioms we use for that proof are true, and so on, in an unending loop.
So, instead, we typically "perceive" facts (1) and (2), in some informal way, rather than proving them mathematically. So, for example, the majority of mathematicians believe that our usual rules of proof satisfy property (2), and believe that the axioms they typically work with satisfy property (1). But they don't prove this in the same way as a mathematical theorem.
If we only assume our axioms are not contradictory, that does not mean they have to satisfy property (1). It is possible to have consistent axioms that are, nevertheless, false.
Property (2) is called "soundness" in mathematical logic, and it is easier to justify than property (1), although it still takes some work. The rules of inference, unlike the axioms we use, are context neutral, so we only have to verify property (2) once, whereas we have to check property (1) again for each new set of axioms.
In some cases, there are heuristic arguments that help to show why a particular set of axioms should be "true". Of course, some people accept these arguments more than others do. For example, it is often argued that the Peano axioms are true because, in our usual understanding of the natural numbers, the natural numbers satisfy all the Peano axioms. Similarly, there is a more intricate argument that tries to justify that the ZFC axioms for set theory are true, based on a particular understanding of what sets are.