I am trying to find which has higher probability when extracting from a box without replacement: two balls of the same color, or two balls with different colors.
Let's assume that in the box are $a$ white balls and $b$ black balls.
$$P(\text{distinct color})=P(\text{first white})P(\text{second black/first white})+P(\text{first black})P(\text{second white/first black})$$ $$=\frac{a}{a+b}\frac{b}{a+b-1}+\frac{b}{a+b}\frac{a}{a+b-1}=\frac{2ab}{(a+b)(a+b-1)}$$
$$P(\text{same color})=P(\text{first white})P(\text{second white/first white})+P(\text{first black})P(\text{second black/first black})$$ $$=\frac{a}{a+b}\frac{a-1}{a+b-1}+\frac{b}{a+b}\frac{b-1}{a+b-1}=\frac{a^2+b^2-(a+b)}{(a+b)(a+b-1)}$$ But there is still needed to find which is more probably to happen, the case which I don't know how to proceed. Do I have to solve (I don't know how) for which domain of $a$ and $b$ does the following inequality hold true? $$a^2+b^2-(a+b)< 2ab \equiv (a-b)^2<a+b $$ This is like in the below answer, so I did it correctly, but still that doesn't help me to conclude an answer..
Total number of possible extractions is $\binom{a+b}{2}$
Number of ways to extract:
Probabilities can be calculated as $$\frac{\binom{a}{2}+\binom{b}{2}}{\binom{a+b}{2}}\text{ for same color and }\frac{ab}{\binom{a+b}{2}}\text{ for different colors}$$ Since you only want to know which has the higher probability, you can simply compare $\binom{a}{2}+\binom{b}{2}$ with $ab$ to see which is greater. Dependent on the values of $a$ and $b$, this could go either way.
Here's a chart detailing which outcome is more likely for $a,b\le30$. A plus (+) indicates that one ball of each color is more likely, a minus (-) indicates that two balls of one color is more likely, and the equal (=) indicates that the probability is split 50-50. Notice that they are equal when $a$ and $b$ are consectutive triangular numbers (1,3,6,10,15...).
Generally, when there are approximately equal numbers of each color, then drawing one ball of each color is more likely.