According to the top answer to this question:
Doing mathematics we often have an idea of an object that we wish to represent formally, this is a notion. We then write axioms to describe this notion and try to see if these axioms are self-contradictory. If they are not (or if we couldn't prove that they are) we begin working with them and they become a definition. Mathematicians are guided by the notion but they work with the definition. Rarely the notion and the definition coincide, and you have a mathematical object which is exactly what our [the mathematicians] intuition tells us it should be.
Formalizing our mathematical intuitions seems to be a tricky business, especially since our intuitions are often contradictory themselves, leading to all sorts of puzzling veridical paradoxes. Additionally, Gödel has shown that it can’t be done in a way that is both consistent and complete, so when we do find a non-contradictory formalization, we have to sacrifice completeness.
But what if we give up consistency instead? Inconsistent systems rather than consistent ones might allow us to formalize our (often inconsistent) intuitions more realistically, if also less usefully.
Unfortunately, the principle of explosion seems to entail that such a system is basically meaningless as every statement would be both true and false. However, there might be some way around this. For example, we could restrict the rules of logical inference in a way that prevents the principle of explosion. Or we could restrict all proofs to below a certain length (corresponding to the limited number of intuitive steps that a person can hold out in one’s head at the same time).
Has this been tried before? Could it be enlightening/useful as a model of human mathematical intuition?
NOTE: From a philosophical rather than mathematical standpoint, plenty of religions/systems of thought are happy to sacrifice consistency to accommodate the inherent contradictions within human intuition. Zen Buddhism is probably the most well known example, and Daoism does something similar if less extreme. I was also reading G. K. Chesterton’s book “Orthodoxy” in which he describes his belief system (he is a Christian), and he asserts that full adherence to logic and reason leads to insanity and absurd consequences, and fails to capture the richness of contradiction in thought and reality.
Yes, such systems have indeed been investigated - key terms include "paraconsistent logics" and "relevance logics." Re: sources, Chris Mortensen has written a summary article and a book on the topic, although the latter has some issues (see here).
Another important term here is "dialetheism." Very roughly, paraconsistent etc. logics are paradox-tolerant in the sense that for a theory in such a logic, a mere inconsistency does not imply triviality. Dialetheism is the philosophical stance that there are true contradictions. Graham Priest has written a lot on the topic (see e.g. here).
That said, I'm not really aware of any plausible attempts to get around the first incompleteness theorem this way: I know of no natural candidates for a theory in a paraconsistent logic which is computably axiomatizable, contains $\mathsf{Q}$ as a subtheory (say), is complete, and is plausibly nontrivial. We can get around the second incompleteness theorem in a weak sense, however: Mortensen's book discusses a particular relevance arithmetic which contains classical first-order $\mathsf{PA}$ but whose nontriviality is $\mathsf{PA}$-provable. (Since nontriviality doesn't imply consistency in this context, this doesn't actually violate the second incompleteness theorem.) Another notable application is the ability of paraconsistent logic to make sense of naive set theory; see e.g. here.