A, B and C are all expert doctors. When each of them (individually) gives a diagnosis (in a yes/no question), the chance of accuracy is 90%, or 9/10. In a case where A and B argue for a certain treatment, and C argues against, what is the chance that A and B are correct? Originally I worked out that the answer is 9/10, but when constructing the general formula I realised that if 11 doctors were arguing against 10 doctors the answer would be the same - 9/10 - which is counter-intuitive and wrong, I the realised that the more doctors who are at loggerheads over a question indicates that this question is more difficult than the average one, in which case the chance of accuracy of all doctors goes down. (90% is only the success rate for the average question, not for the more difficult ones.)
My question is: given these facts, is there any way to give a formula which will tell me the probability of accuracy given the number of doctors on both sides of the issue (assuming that all doctors have the same expertise, that they answer 90% of questions correctly)?
If anyone could help me I would appreciate it!
Taking a Bayesian approach there is
a random variable $C$ for whether or not the patient has a condition, in which case $C$ takes the value $1$, or hasn't, in which case $C$ takes the value $0$.
There are also random variables $N_1$, the number of doctors saying the patient has the condition, and $N_0$ the number of doctors saying the patient hasn't.
Also
Then using Bayes Theorem on the first line, the Theorem of Total Probability on the second line, and cancelling the $\left(\matrix{n_0+n_1\\n_0}\right)$s on the fourth line
$$\begin{align} P(C=0|N_0=n_0\wedge N_1=n_1) &= \frac{P(N_0=n_0|C=0)P(C=0)}{P(N_0=n_0)} \\ &= \frac{P(N_0=n_0|C=0)P(C=0)}{P(N_0=n_0|C=0)P(C=0) + P(N_0=n_0|C=1)P(C=1)} \\ &= \frac{P_B(n_0,n_1,p)P(C=0)}{P_B(n_0,n_1,p)P(C=0) + P_B(n_0,n_1,1-p)P(C=1)} \\ &= \frac{p^{n_0}(1-p)^{n_1}P(C=0)}{p^{n_0}(1-p)^{n_1}P(C=0) + (1-p)^{n_0}p^{n_1}P(C=1)} \\ &= \frac{P(C=0)}{P(C=0) + (1-p)^{n_0-n_1}p^{n_1-n_0}P(C=1)} \;\;\;(\dagger)\end{align} $$
This is a formula involving the probabilities of whether the patient has the condition or not, and the difference between the number of doctors who disagree, so the total number of doctors doesn't matter.
Then assuming we don't have any prior reason to believe that a patient has the condition or not, i.e. $C=0$ or $C=1$ is equally likely, setting $P(C=0)=P(C=1)=\frac12$, we find that $$ P(C=0|N_0=n_0) = \frac{1}{1 + (1-p)^{n_0-n_1}p^{n_1-n_0}}$$
With $p=0.9$, this gives $P(C=0|N_0=1\wedge N_1=0) = 0.9$, and $P(C=0|N_0=1\wedge N_1=1) = 0.5$, as you'd expect. If A and B argue for a certain treatment, and C argues against, then $P(C=0|N_0=2\wedge N_1=1) = 0.9$
If talking about probabilities for whether a patient has a condition or not sounds rather hand-wavy, you can think in terms of classical probability, where there isn't a random variable $C$. The patient either has the condition or not: there's just a fixed but unknown parameter $c$ to say whether they have it. The observed events are: the number of doctors $N_1=n_1$ saying $N_2=n_2$ and the number of doctors $n_0$ saying $c=0$. You would calculate the likelihood or conditional probability of seeing $N_1=n_1, N_2=n_2$ given $c=1$ and the probability of seeing $N_1=n_1, N_2=n_2$ given $c=0$. We have two different possible models, each with a different parameter, one with parameter $c=0$ and one with $c=1$, and we choose the parameter with the maximum likelihood. This doesn't help you very much in terms of getting a probability! And nor should it -- the classical approach holds that the patient either has the condition or doesn't, and so can't give you a probability.
$(\dagger)$ I have a feeling there should be an easier way of getting to this result: if someone can spot it please let me know!