How exactly are modern settings for homotopy theory like simplicial sets or cubical sets more powerful or useful than classical settings like mere topological spaces, or CW-complexes?
How exactly do such 'combinatorial objects' capture all information of interest up to homotopy? What are some down-to-earth examples illustrating the advantages of preferably working with these 'combinatorial objects'?
How essential is it for the sake of doing homotopy theory, that these modern / abstract homotopy theories are formulated in categorical settings, the way they are?
All in all, what's the storyline that leads one to these contemporary categorical abstract homotopy theories, starting from basic classical notions like the fundamental group of a topological space?
“More powerful” and “more useful” are not necessarily claims that most mathematicians would defend. It seems to me that homotopy theorists generally think in terms of some informal notion of space up to homotopy which is certainly more like a topological space than like a simplicial set. Working with simplicial sets is in large part a technical trick to do homotopy theory in terms of a much simpler category than topological spaces. Simplicial sets, as a presheaf category, are abstractly almost exactly as well behaved as plain sets, whereas topological spaces are harder-for instance it takes a lot of technical work to get internal homs, which are nice.
As for how simplicial sets “capture all of homotopy theory”, you should think of a simplicial set as being essentially the same as its geometric realization, which is a CW-complex. So the main trick is to show that every CW-complex is homotopy equivalent to the geometric realization of some simplicial set, which isn’t too hard-the singular simplicial set will do nicely. Thus it was established early on that CW complexes and simplicial sets have the same homotopy theory-and simplicial sets are vastly better behaved categorically than CW complexes.
I’m not sure how to respond to the third paragraph, for reasons suggested in the comments. Category theory was invented in large part to serve algebraic topology, when the latter subject was a teenager. “Essential” is a hard word to justify, but essentially all algebraic topology that’s actually been done since the 1940’s has been at least moderately categorical in perspective.
The storyline is perhaps easier to see from a starting point of homology. The homology of a simplicial complex is quite computable, whereas singular homology is unusable for concrete calculations, just good for some theory. So people wanted to think about simplicial complexes rather than general spaces when possible. Passing to CW complexes makes this work a bit better, but there’s still some tension since a space doesn’t come with a distinguished CW structure and of course not every space even admits one. It is basically fair in this context to think about a simplicial set as “a space with a given CW structure”, though of course the details aren’t precisely the same.