I'm looking to create a variant of holdem allowing 6 and 7 card hands in special situations. Namely, 3 pair, 2 set, and quads over set (a full mansion).
My probability skills are rusty, but I'm still able to break down the problems, I think!
I've started trying to calculate the odds of 3-pair so I can find where to rank the hand. I'm using the probability tree method. The branch at the top is clear, pocket pair vs non pocket pair.
When you go down the pocket pair side (0.059), the board must also 2 pair to have a 6 card 3 pair hand.
There are 66 combinations (12 choose 2) of the other than our current pocket pair pairs that the board can make.
The odds of a single combination of such a board would be, I believe, (4/50) * (3/49) * (4/48) * (3/47) * (40/46) = 2.26548e-05
The reason for 40 is because you can't create a set of any of the 3 pairs with the last card or else you'd create a full house, so you need to subtract those 6 remaining cards.
I'm struggling to remember how exactly to use this value. I know it's not just multiplying by 5! due to the fungibility of each paired card.
I think I need to use permutation with repetition? That would mean 5! on the numerator, then divided by (2! * 2!), coming out to 30.
Meaning the final calculation for the odds of there being 3 pair when you are holding a pocket pair to be:
2.26548e-05 * 66 * 30 = 0.044856504
On the other side of the tree, where you don't have a pocket pair (.941), the board must pair both your pocket cards and also pair itself for you to have 3 pair.
There are 11 other pairs the board could make besides the 2 cards you're holding.
The odds of a single permutation of such a board is:
(3/50) * (3/49) * (4/48) * (3/47) * (40/46) = 1.69911e-05
Using the same permutation with repetition formula, it should be (5! / 2!) or 60.
So the odds of getting 3 pair with a mixed hand would be 1.69911e-05 * 11 * 60 = 0.011214126
Combining both sides of the tree, we get
pocket pair = 0.059 * 0.044856504 = 0.0026465337
mixed hand = 0.941 * 0.011214126 = 0.0105524926
For a combined odds of 0.0138606597 or 1.386%.
That actually would rank higher than a full house (2.6% chance) which is surprising to me. I figured this would rank much lower closer to a straight or a set. Of course, that assumes I didn't mess up my math somewhere! Thanks for reading and please help me correct my mistakes :)
Since you chose a somewhat roundabout way to do the calculation, let’s do it again in a slightly more direct way to check your result. There’s no need to treat the two pocket cards any differently than the five board cards – all you want to know is the probability for seven cards to contain three pairs, without any distinction among the cards.
We need to choose $3$ out of $13$ ranks for the pairs, $1$ more rank out of the remaining $10$ for the singleton, $2$ out of $4$ cards for each paired rank, and $1$ out of $4$ cards for the singleton, so the probability is
$$ \frac{\binom{13}3\binom{10}1\binom42^3\binom41}{\binom{52}{7}}=\frac{2376}{128639}\approx1.85\%\;, $$
slightly higher than your result but still lower than you expected.
Two errors kept you from reaching this result. One is where you say “the board must pair both your pocket cards and also pair itself for you to have $3$ pair”. That’s not true – the board can pair one of your cards and add another two pairs. Since we can choose $1$ of $2$ pocket cards to pair, $1$ of the remaining $3$ cards of that rank, $2$ out of $11$ remaining ranks for the other pairs and $2$ out of $4$ cards of those $2$ ranks, the probability for this is
$$ \frac{\binom21\binom31\binom{11}2\binom42^2}{\binom{50}5}=\frac{297}{52969}\approx0.56\%\;. $$
The other error is in your addition of $0.0026465337$ and $0.0105524926$, which yields approximately $1.32\%$, not $1.386\%$. If you correct those two errors, your approach also yields
$$ \frac1{17}\cdot\frac{\binom{12}2\binom42^2\binom{10}1\binom41}{\binom{50}5}+\frac{16}{17}\cdot\frac{\left(\binom{11}1\binom31^2\binom42\binom41\binom{10}1+\binom21\binom31\binom{11}2\binom42^2\right)}{\binom{50}5}=\frac{2376}{128639}\approx1.85\%\;. $$