Problem: Consider players $A$ and $B$ rolling a 20-sided die. Player $B$ is allowed to re-roll. Assume that player $B$ will re-roll a single time if his first dice roll is $\leq 10$, and will not re roll if their first dice roll is $> 10$. The comparison is conducted between the last roll of $B$ (not the max roll of player $B$) with $A$. Let $RR$ and $NR$ denote the event of player $B$ rerolling and not rerolling. I want to find the probability of $NR$ occuring given that the first player rolls a higher number than the second player using a combinatorics approach.
Denote $A$ and $B$ as the random variables for the outcome of $A$ and $B$'s tosses. First, I computed the case where player $B$ isn't allowed to reroll. The result of this case is used later. For this case where $B$ can't re-roll, consider the possibilities in which $A > B$ is satisfied:
For $B = 1$, there are 19 possibilities for $A$.
For $B = 2$, there are 18 possibilities for $A$.
... For $B = 19$, there is 1 possibility for $A$.
So the number of possibilities for $A > B$ is $\frac{19 \cdot 20}{2} = 190$ when $B$ isn't allowed to re-roll. .
As stated previously in words, I want to find $$ P(NR | A > B) $$
combinatorically. From a pure probability approach, I found that the probability is $\frac{9}{28}$ and confirmed this via monte carlo using 10 million trials.
At first, I sought the number of cases such that $A > B$. To do this, I considered the disjoint events $NR$ and $RR$. For the event $NR$, i.e., where player $B$ rolls once and his outcome is $> 10$, there are 45 cases where $A > B$. For $B = 11$, we have $9$ cases, ...., for $B = 19$, we have 1 case. So we have $\frac{9 \cdot 10}{2} = 45$ cases.
Then I considered the event $RR$, i.e., where player $B$ rolls a $\leq 10$ on the first toss and rolls again. This is effectively the same problem as the first problem I did in this post. Here, player $B's$ outcome can be $\in \{1, \ldots, 20\}$. As we saw at the beginning of this post, there are $190$ cases in which $A > B$.
So my first inclination was that $$ P(NR | A > B) = \frac{45}{45 + 190} = \frac{45}{235} $$
where the denominator is the sum of the number of $A > B$ cases in the two disjoint events $RR$ and $NR$.
But this probability is smaller than $\frac{9}{28}$, so it seems I may have overcounted or possibly undercounted somewhere.
It seems it is the the $RR$ case in which I overcounted the number of cases with $A > B$. In particularly, it seems that I may be duplicately counting the cases where $A > B \cap B \in \{11, \ldots, 19\}$, since I already counted this in the event $NR$. But it's not clear to me why this is overcounting when I defined $RR$ and $NR$ to be disjoint. Nevertheless, if we operate on this assumption that this did in fact overcount the cases where $B \in \{11, \ldots, 19\}$, then that means the number of cases where $A > B$ should be reduced by 45.
So now our result is
$$ P(NR | A > B) = \frac{45}{45 + 190} = \frac{45}{190} $$
But this result is still not correct, and does not match the pure probability approach and monte carlo. Originally, I thought this problem would be simple from a combinatorics perspective, but now I am confused why my thought process doesn't work.
What is wrong with my approach?
First, I will provide the way I would go about computing the desired probability without regard to the method of solution. The key is to use Bayes' theorem: $$\Pr[NR \mid A > B] = \frac{\Pr[A > B \mid NR]\Pr[NR]}{\Pr[A > B]}.$$ Given that no reroll occurred, the set of outcomes for $B$ is $\{11, 12, \ldots, 20\}$, each with probability $1/10$. If $B = b$, then $\Pr[A > B \mid B = b] = \frac{20 - b}{20}$, hence $$\Pr[A > B \mid NR] = \sum_{b=11}^{20} \Pr[A > b]\Pr[B = b \mid NR] = \sum_{b=11}^{20} \frac{20 - b}{20}\cdot \frac{1}{10} = \frac{9}{40}.$$ The probability of the event $NR$ is simply $\Pr[B > 10] = \frac{1}{2}$.
The denominator, which is the marginal probability of $A > B$, is not $19/40$, which was the calculation when $B$ was not allowed to reroll. By the law of total probability, $$\Pr[A > B] = \Pr[A > B \mid NR]\Pr[NR] + \Pr[A > B \mid RR]\Pr[RR].$$ The first term is the same as the numerator we calculated earlier. For the second term, the probability $$\Pr[A > B \mid RR] = \sum_{b=1}^{20} \Pr[A > b]\Pr[B = b \mid RR] = \sum_{b=1}^{20} \frac{20 - b}{20} \cdot \frac{1}{20} = \frac{19}{40},$$ which is the original calculation--this is because if $B$ did reroll, then the outcomes and their probabilities are the same as if $B$ was never allowed to reroll in the first place. But now we have to also take into account $\Pr[RR] = \frac{1}{2}$. So our desired probability is $$\Pr[NR \mid A > B] = \frac{\frac{9}{40} \cdot \frac{1}{2}}{\frac{9}{40} \cdot \frac{1}{2} + \frac{19}{40} \cdot \frac{1}{2}} = \frac{9}{28}.$$
The problem with your approach is that your outcomes are not equiprobable, so you cannot simply add the number of outcomes as you did. You have $45$ outcomes in which $A > B$ given $NR$, but these outcomes are out of a total conditional set of outcomes of size $20 \cdot 10 = 200$, whereas in the case that event $RR$ occurred, the total number of outcomes is $20 \cdot 20 = 400$. Therefore, your calculation must be modified to weight the outcomes accordingly: $$\frac{\frac{45}{200}}{\frac{45}{200} + \frac{190}{400}} = \frac{9}{28}.$$ In essence, you have intuitively applied Bayes' theorem in your reasoning but you applied it incorrectly.
To see how you would do this calculation if $\Pr[NR] \ne \Pr[RR]$, it is immediately obvious with Bayes' theorem. But your "combinatorial" or enumerative approach requires more subtlety. The reason your approach could even be adjusted/weighted at all was because it so happened that the criterion for $B$ to reroll is such that the probability of a reroll is equal to the probability of no reroll. So we give an example where this is not the case by modifying the criterion.
Suppose $B$ decides instead to reroll if the first roll is less than or equal to $7$. Then in the case where there is no reroll, there are $13(12)/2 = 78$ desired outcomes where $A$ outrolls $B$, out of a possible $13(20) = 260$ outcomes. If there is a reroll, then as before, there are $190$ desired outcomes where $A$ outrolls $B$ out of a possible $20(20) = 400$ outcomes. But because the probability of reroll is $7/20$ and no longer the same as the probability of no reroll which is $13/20$, the probability of no reroll given $A$ outrolled $B$ is now $$\frac{\frac{78}{260} \frac{13}{20}}{\frac{78}{260} \frac{13}{20} + \frac{190}{400} \frac{7}{20}} = \frac{156}{289}.$$