I'm a graduate mathematics student and I've taken several courses on statistics, probability theory, stochastic processes and machine learning.
In all the textbooks I consulted and all the classes I've taken so far there is a great amount of mathematical detail and explanations regarding the simple concepts of the subject at hand. Specifically in statistics and machine learning there is often a lot of text on the description of the problem.
However, as soon as the problem is given they go straight to the model distributions. E.g. in our course on insurance mathematics estimating the insurance premium was done using a Poisson-Gamma distribution approach. But while there was a great deal of discussion on the (lack of) importance of the horsepower of the insured car, there was no explanation for why we would use the Poisson or Gamma distributions in our model.
Everywhere I look it is just assumed I already know which distribution is best fitted to model a specific problem.
Do you know any literature that explains in detail what the variety of modelling distributions we have represent and when and why to use them?
This is the book you are asking for:
Statistical Distributions 4th Edition, by Forbes, Evans, Hastings and Peacock, 2011, freely available here.
Then, I would add also: Ross, Introduction to Probability Models (7th ed. 2007) as a good reference point.
These books would tell you "here are the distributions we often use, when and why" but I think you are interested in the inverse problem: "Given a setting, what distribution shall I use to model it best?". To my knowledge no such book exists; modeling requires intuition, experience and is data-driven too.