I've been asking myself this and other questions in the field of philosophy of mathematics. Could we, if we were isolated from any kind of sensory experience, be able to learn mathematics?
Also, what does it take to learn math?, is there a 'module' or a 'structure' (like the one theorized by Noam Chomsky in his studies on linguistics) in our brain that lets us apprehend mathematics?, do we need a language (be it natural, be it symbolic) to learn mathematics?
PD: I'm a beginner to SE, if this question if off topic please do tell and do recommend me where should I ask it.
I think this is a very interesting question, which is hard to formulate precisely. It's also controversial. You might be interested in Misha Gromov's theory on the "Ergobrain." Here is an attempt at a summery of this idea.
Gromov partly attempts to first pose and then answer questions similar to yours. Roughly speaking, Gromov thinks that much of mathematics that we are capable of is highly dependent on (limited by) the structure of how our minds think. He defines the "ergobrain" as a system that takes raw information and tries to form structures from it. I am not an expert in category theory but to me his idea is to try to categorize the way systems can build mathematical-looking structures. The ultimate conclusion is that, at least when it comes to human minds, we think we know what mathematics is but in reality, we don't. Putting it more mildly, it takes a lot of effort to transcribe our thoughts into mathematics language. This is not saying that mathematics lacks rigor. The issue is that we cannot perfectly dissociate ourselves from our sensory experience which corrupts the fact that mathematics is fundamentally rooted in axioms. We think we know that $2+2=4$ based on our daily experience but in reality, proving such a statement from fundamental axioms is highly nontrivial but doable.
In other words, our physical confirmation that $2+2=4$ falls woefully short of rigorously defining what it means mathematically. Here's a typical example of this, where one wants to prove that $\sqrt{5}$ exists (there is some real number equal to $\sqrt{5}$), a seemingly ridiculous question for our sensory experience (it's somewhere between 2 and 3!). Yet, you do not need to know Peano arithmetic to teach a baby how to add numbers. This is precisely the distinction between the ergobrain and mathematics. In fact, it's not unreasonable to guess that the baby, after enough exposure to the world around her, will develop her own sense of $2+2=4$. It already takes a leap of faith to assume that two apples plus two apples equals four apples carries over to oranges, clouds and sand. Now you're unconciously developing the concept of equivalences, isomorphisms etc. So, that must be something innate to your mind, the capacity to declare equal and not equal. Unless you axiomatize it though, it's still just sensory, not mathematics.
Other examples of this include Boltzmann's original definitions of what entropy is, based on somewhat loose physical ideas and principles (here are his thoughts on this). A rigorous mathematical definition is considerably more difficult to formulate and took another half century until Kolmogorov and others provided rigorous foundations for this field. I hope all this doesn't sound disparaging. Afterall, great ideas are not born overnight, especially in mathematics.