Consider a game where you gamble with money. Several outcome may exist with different probabilities of occuring where you win or lose varying amounts. Let $X$ equal the amount of money retaind when you're done playing. Assume that $P\{X>0\}>1/2$ and that $E[X]<0$. So that essentially means that if you play the game many times you're going to win more times than you lose but you're still going to end up losing money since some loses may be of a great amounts of money. If you're deciding upon whether you should play one game or perhaps even several games then probability alone is not necessarily enough to go on when it comes to making this decision because you might have to take into account the risk of losing a great amount of money. This risk varies of course depending on how much one might lose. So is it possible to numerically measure risk in a way such that risk and probability both can be taken into account upon making a decision? Is there any established mathematical theory or branch of probability that deals with this? I'm asking because I just stumbled upon an exercise which yielded a similar result and it made me think.
Edit $1$: I'm studying elementary probability so I don't think this question will be answered in my current book, thus I am asking it here.
Edit $2$: I found something on google called game thoery, is that it?
I think you're thinking of decision theory, which is often considered a subfield of economics and statistics.
One simple way of thinking about this problem is to define preferences over outcomes. If those preferences are well-behaved enough (for instance, if they obey the von Neumann-Morgenstern axioms), then those preferences can be represented by a utility function over outcomes.
If the space of outcomes is the real line, one simple utility function that you might want to use to capture preferences that exhibit risk-aversion is the logarithmic function. With these preferences, you might say you'd be willing to take a bet that gives a payoff of $X$ (which is a random variable) if
$$ \mathbb E[\log(W+X)] \geq \log W $$
where $W$ is the amount of money you start with.
These issues are also relevant in game theory, which you can loosely think of as multi-person decision theory.
Kreps has a great monograph which I think serves as an excellent introduction to decision theory. If you're feeling ambitious, you might also want to read Savage.