Calculate continuous probability distribution of true winrate based on given sample

Question

Basically, I was wondering if you have a sample of N games and have won X number of games (binary outcome), you would be able to accurately determine the mean and variance for the "true WR." This is complicated by the fact that this WR must lie between 0-1. For example, if you would win 5/5 games, the mean expected WR would not be 100%, but would be a bit below that. I feel like it is possible to calculate a continuous probability distribution to determine the WR and variance (the probability of 5/5 would be 1.0 for 100%WR, .99^5 for 99%WR, etc, down to 0.0 for 0%WR, which should produce a curve of finite volume which can be scaled into a probability distribution). This should work for any sample/number of games won. I just don't know if this method is actually feasible to calculate mathematically or programatically or if I should just give up and go for some other simpler approximation.

Answer 1

You could think about this in terms of Bayesian statistics. Suppose that $X_1,X_2,...$ are i.i.d. $Bernoulli\left(\rho\right)$ with unknown $\rho$. You start by imposing the prior distribution $\rho \sim Beta\left(\alpha,\beta\right)$, which has support in $\left[0,1\right]$. If you do not have any prior knowledge, you can try a non-informative prior with $\alpha=\beta=0$, otherwise you can use some small initial value. Then, the posterior distribution of $\rho$ given $X_1,...,X_k$ is still beta with parameters $\alpha_k = \alpha + \sum^k_{i=1} X_i$ and $\beta_k = \beta + \sum^k_{i=1}\left(1-X_i\right)$. Basically the parameters count the numbers of successes and failures, respectively. In this way you get an entire probability distribution for $\rho$ which changes over time (and eventually concentrates on the true win rate) as more observations are collected.

In fact, the wording "mean and variance for the true WR" only really makes sense in a Bayesian world, where any unknown value is modeled as a random variable. Outside Bayesian statistics the true win rate is a fixed number -- you may not know it, but it is still fixed.

Note that, if you win the first five games as in your example, the posterior parameters will be $\alpha+5$ and $\beta$, so the posterior mean of $\rho$ will be $\frac{\alpha+5}{\alpha+\beta+5}$, which for small $\beta$ has exactly the behaviour that you want (estimating the win rate at slightly under 100%).