The way I work on homework questions, especially for analysis and topology, might be a little different (or maybe not). I would remember the questions and think of them when I run, take a shower, and during other periods when I don't need to use my brain; later I will write out the solution without much thinking. This typically works well even on harder questions that my professor expects more than two hours to solve. However, sometimes I forgot about what I thought before, and my notes for insights, which are random diagrams or phrases on a scratch paper hidden in a stack of papers, are normally either not found or incomprehensible, so I have to rethink the questions. When I actually write out the proof, it is very readable, but as an undergraduate, I have many other time commitments, so I don't have enough time to completely elaborate or type out what I'm thinking. In that case, how can I record my thoughts quickly so I will be able to reproduce them?
Also, sometimes I would try different approaches, and typically most of them do not work, so how can I record these attempts and not go into loops? What is frustrating is sometimes I go back again and again on a "branch" that is seemingly close to the answer but is very unrelated and other times I spent a too short amount of time on something that is close to the solution. Is there a way to prevent this?
For mathematical research, I would imagine the complexity dramatically increases. For my research project (as an undergraduate working in mathematical optimization), my professor typically introduces to me some very nice lemmas, which help me to prove the optimal goal. However, if I'm working on complicated questions by myself, how can I keep track of all the methods I tried and not worry about forgetting about them the next day? Of course, typing everything out is a solution, but how exactly can we type something that is just vague intuition in our brain instead of proofs for lemmas and theorems?
I know how you feel; I had the same issue with my research notes in physics. Of particular difficulty is that in physics, many results are attained by pure intuition, and what we refer to as "proof by Mathematica", which involves checking my "theorem" for many possibilities. Almost any formal proof that I write down comes after I checked the result on Mathematica. Here is how I keep track of my "guesses" for a theorem:
I state the problem clearly: e.g. "Find the Schmidt decomposition for symmetric vectors generated from $SU(d)$ for all $d$" (this is what I am currently working on).
An iPad has the advantage that I can screenshot/copy a result, theorem, equation, etc. from published works. So when I try to solve this problem for general $d$, I first find a source that proves the simpler/similar case ($d=2$) and annotate to provide useful commentary. This page is labeled accordingly.
Then, I might try to find the formula for $d=3$: I label my guesses by how I arrived at them, and then check on Mathematica if they are correct; either way, I add commentary on the result in a different color. I have found that this really helps in figuring out what's going "wrong" with my guesses.
Once I obtain the "correct" theorem, it is often easy to guess the correct theorem for general $d$. This essentially leaves me with a new problem: proving that this theorem is true. Similar labeling and color-coding is used for different ideas, attempts, and commentary when proving the theorem.
This is my first answer to a soft question; feel free to give advice on how to improve my answer!