Suppose there are $R$ number of red balls, $B$ number of black balls, and $W$ number of white balls in an urn. The total number of balls are $N$, thus, $N = R + B + W$. I draw $M$ number of balls from the urn, where $N \geq M \geq B$. The probability of drawing all the black balls, but none of the white balls follows the multivariate hypergeometric distribution:
$$P = \frac{\binom{N-B-W}{M-B} \cdot \binom{B}{B}}{\binom{N}{M}}$$.
Now I want to find the value of $M$ that maximizes the probability $P$. I was initially thinking of taking the derivative of $P$ with respect to $M$ by transforming the binomial coefficients into gamma functions, but it was too complicated. Any suggestion?
Since you MUST choose all black balls, we can ignore the second factor in the numerator. Now, let $\delta = M-B$, we can then re-express the probability as:
$P=\frac{R \choose \delta}{N \choose B+\delta}=\frac{R!(B+\delta)!(W+R-\delta)!}{\delta!(R-\delta)!N!}$.
When $\delta=0$ we get $P_0 = \frac{B!(W+R)!}{N!}$. If we increase to $\delta=1$, the numerator will change by a factor of $\frac{B+1}{W+R}$ and the denominator will change by a factor of $\frac{1}{R}$. Putting this together, we get a total change factor of: $\frac{R(B+1)}{W+R}$. Increasing $\delta$ by 1 to get $\delta=2$, we will further change the probability by a factor of $\frac{(R-1)(B+2)}{2(W+R-1)}$.
As long as each of these factors are $\geq1$, we should increase $\delta$. Therefore, we want to find $\max\delta: \frac{(R-\delta+1)(B+\delta)}{\max\{1,\delta\}(W+R-\delta+1)}\geq1$. If we solve the continuous relaxation for the case where LHS=1, we get the equation $(R-\delta+1)(B+\delta)=\delta (W+R-\delta+1)$ where $\delta \in [1,R]$
Expanding the terms, we get: $RB+R\delta-\delta B-\delta^2+B+\delta = W\delta+R\delta-\delta^2+\delta$.
Cancelling like terms on both sides, we get the simplified equation:
$RB-\delta B+B = W\delta$, where we get $\delta = \lfloor \frac{B(R+1)}{W+B}\rfloor$ as the optimal value.
As an example, if $W=5, B=7, \text{and}\; R=4$, we get $\delta = \lfloor \frac{7\times 5}{5+7} \rfloor = \lfloor \frac{35}{12}\rfloor = 2 $ so you would choose $M=B+2 = 9$ balls.