I've found a method for integration of $\frac{1}{e^x +1}dx$. However none of the information I found explains the intuition. It might be clear for most people, but I was hoping that somebody could expand on it.
The problem of trying to u-substitute $\frac{1}{e^x +1}dx$ becomes evident.
Since $\frac{1}{e^x +1}dx$, where I substitute $e^x$ for u, gives me:
$$\int\frac{1}{u}\frac{du}{e^x}$$
Which still leaves me with $e^x $.
The method that solves it is by expanding the numerator with $e^x - e^x$ which gives us the integral:
$$\int \frac{e^x + 1 - e^x}{e^x +1}dx$$
I have seen that it works. But what compels somebody to expand the numerator with $e^x - e^x$? What benefit does a mathematician intuitively see with this? How could I be sure that it just doesn't give me back the initial integral:
$$\int\frac{1}{u}\frac{du}{e^x}$$ ?
Various integrals - or really, a number of problems in math (I've seen it used in real analysis for stuff involving limits by the $\epsilon - \delta$ definition) - can be solved by this technique of adding and subtracting the same thing from a quantity. For example,
$$e^t = e^t + x - x$$
This is because $x-x$ (anything minus itself) is $0$. So adding and subtracting something from itself explicitly like this is fundamentally no different from adding $0$ to something, and thus perfectly valid - if pointless at face value. But sometimes this adding and subtracting suffices in giving us a manipulation that is more handy. In this case, we see that we can split up the fraction after this manipulation:
$$\frac{1}{e^x + 1} = \frac{e^x + 1 - e^x}{e^x + 1} = \frac{e^x + 1}{e^x + 1} + \frac{-e^x}{e^x + 1} = 1 + \frac{-e^x}{e^x + 1}$$
Notice how this might be easier to integrate, namely because of a $u$-substitution of $u = e^x+1$? This manipulation lets us do that!
I'm not sure if there is some sort of underlying intuition, necessarily, beyond trying to find a method which works. Some problems are a matter of trial and error to find something which works.
Sure, sure, in school you may often know ahead of time the method you need to use because you might be covering related material at the time and it's like "well, why would the homework be on something unrelated to the material, so obviously I'm expected to use this method that we're covering." But if a problem faces you out of the blue, say in competition mathematics or in the real world, you have less guidance and so trial and error might be necessary.
Of course, there are other ways to solve this particular integral, as was touched on in two other answers, you need not limit yourself to one method of problem-solving.
It's just a nice little tool to have in your arsenal, one that should be kept in mind because despite its simplicity it is deceptively helpful in some cases. I don't think there's an underlying reason as to why or when it should be used other than "it works," though.