An argument came to my mind that seems really mind-blowing and I haven't found it anywhere. Here's how it goes:
We call a formal system F embodied in classical logic a foundation of mathematics when each and every statement of mathematics is translatable into its language and interpretable accordingly.
By Gödel's second incompleteness theorem, we know that any foundation of mathematics cannot prove its own consistency because it necessarily should include arithmetic.
If we came up with a mathematical proof of the consistency of F, then by definition of F we would be able to mimic that proof inside F, contradicting Gödel's second theorem, proving F to be inconsistent, so F is useless.
In other words, if F is consistent, not only are we not able to prove its own consistency inside F, we are not able to prove its consistency using tools beyond it.
Therefore, if a foundation F really is absolutely consistent, there is no way to know it is.
Is this a good argument?
I think the argument leaves out some other methods of proving the consistency of F, methods that are not at all mathematical. But knowing that F is itself a mathematical object, what else can we use to study it?
This kind of question is dicsussed and (for me) well answered into:
In a nutshell, every "absolute" foundational system F will be obviously open to some sort of "skeptical challenge": any conceivable system will be based on some assumptions, and it is always possible to ask for the justification of these assumptions.
Regarding an "absolute" consistency proof of $\mathsf {PA}$, the natural numbers sequence $\mathbb N$ is a model of it and is a "standard" result of logic that a theory having a model is consistent.