Say you have a two-player game and three, fixed-but-different skill players: A, B and C. Player A only plays against C and player B also only plays against C. You want to to determine if player A is better than player B.
Say A wins 60 of 400 games with C while B wins 120 of 400 games with C. Clearly, in this case, one can say B is better than A.
But if the results are closer, say: A wins 150 games out of 400 with player C. B wins 165 of 400. With what confidence can we still say that B is likely to actually be better than A?
I think a binomial test would be appropriate if the two players I'm interested about were directly playing against each other. If A wins 175 out of 400 versus B, we can be >95% certain that B is better than A (98.6% certain). But I'm not sure how to apply that if the players can only face a common opponent but not each other.
Use Proportion Differences Hypothesis Testing
p1 = Proportion of games A wins against C
p2 = Proportion of games B wins against C
H0 : p1-p2<=0
H1 : p1-p2>0
Pooled Sample proportion = (p1*n1+p2*n2)/(n1+n2)
p1 = 150/400 = .375
n1 = 400
p2=165/400= .4125
n2 = 400
p = (150+165)/800 = .39375
SE $= \sqrt{p*(1-p)*\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}$
$ = \sqrt{.39375*.60625*\left(\frac{1}{400}+\frac{1}{400}\right)} = .03455$
Z statistic = (p1-p2)/SE = (.375-.4125)/.03455 = -1.085
level of significance is$\alpha = 0.05$
P(z<=-1.085) = 0.13889
p-value = 0.13889
Since the P-value (0.13889) is greater than the significance level (0.05), we cannot reject the null hypothesis and conclude that player A is less talented than player B.
In other words we can be 86.11% certain that player B is better than player A Thanks
Satish