How to know when to give up on a ${}$ math problem, which may not have a solution?

Question

When you're working on problems from textbooks, you know that the problem has a solution, because the author claims that it has.

But when you're doing original research, you generally don't know whether what you're looking for exists. You might spend weeks on it, but there might simply not be a solution at all.

How do you decide to give up on such a problem?

Answer 1

That's impossible for anyone else to know. Just look at the 350 year long history of Fermat's last theorem. Until the mid 1990's, no one knew whether it was true (and not for lack of trying). If Andrew Wiles hadn't succeeded in the end, he would've spent almost a decade locked up in his own attic for nothing. He didn't give up. Should he have? In all likelihood, many would have advised him to give up (and I'm sure some did), but that's not the same thing.

It's not about whether you objectively should give up at any given point; it's always about the risk and whether you personally think it's worth it. And you are the only one who can answer that.

Answer 2

Via the Halting Problem and Incompleteness Theorem, in some sense there isn't a perfect answer to your question. That said, there might be lots of good answers depending on your goals. I'll answer the related question, "How do you decide [when] to give up on such a problem?"

• Never. Pause problems rather than throwing them in the bin. People learn over time, and there is no telling what new fact, tidbit, or insight is going to make the difference in your mental model of a previously hard problem.
• When you find something more productive. Talking to colleagues or attending conferences exposes oneself to new ideas. When combined with your own personal worldview and experience, this can open up a flurry of productive research topics. If your goal is to quickly churn out new work, these paths where you can see closer to the end might be a better idea than your current roadblock.
• Whenever you find something more interesting. Life isn't all about productivity. Study what makes you happy, and when that fails study something else. Eventually you'll find a project that is also productive, and you won't have beaten yourself up to find it. Most of the math professors I know use some variation of this strategy.
• Never. I know I listed this twice, but this can be important for another reason. Work on hard problems often turns up other intriguing results. Another answer by @Arthur mentioned Fermat's Last Theorem, and that's a great example of entire fields of math being discovered just to create a single proof. I have a pet number theory problem I reexamine periodically that always has something new going on.
• Often. Staring at the same problem without progress is a waste of time and drains your mental resources. Unless you're really making headway toward something, take a break every half hour or so and go for a walk or find somebody to talk to.
• When progress stalls. On too many occasions to count, problems I thought would be fruitful did not have clean answers that I could find. After exhausting all available resources, I filed away my work to pick them up at a later date. With additional knowledge and experience I've since solved some of them, proven others can't be solved, and made absolutely no headway with a substantial majority of those files.

Everybody has their own tolerance for work without results, but mine is very low if I'm not at least learning something new or making some small discovery in the process. Other people suggest having a single project at a time (or a major project with large impact and a minor project to get a guaranteed paper out), but I think overall productivity is higher in creative disciplines like math when you can switch to whichever idea you currently have the most insights into, at the expense of each paper taking longer from start to finish.

Answer 3

This is where the Art of mathematics takes over from pure logic. You may 'know' intuitively there's an answer and spend your career looking for it. You'll get either a Nobel Prize or nowhere! Probably better to work on other stuff as well though.