Confidence two biased dice are the same?

Question

I have 2 biased dice (die 1 and die 2) and I would like to calculate the % confidence they are the same (or different), given $n_1$ rolls of the first die and $n_2$ rolls of the second.

Conceptually I'd expect that initially the confidence that they were the same (or different) would be $0$, and as $n_1$ and $n_2$ increase towards $∞$ the confidence would approach $100\%$ that they are the same (or different).

It's relatively trivial to generate a distribution curve of the probability of rolling a specific value on each die, but it's unclear to how how to compare 2 distribution curves (one from each die) to determine the confidence that they are the same or not.

Answer 1

Let $p_{k}$ denote the parameters of the $k$-th die (a vector of probabilities corresponding to each side) and let $\hat{p}_{k,n}$ be its sample analogue (sample proportion). A possible measure of similarity between the dice is $$ S(p_1,p_2):=1-\frac{d(p_1,p_2)}{\max_{p,q\in \Xi}d(p,q)}, $$ where $d(\cdot,\cdot)$ is a distance on $\mathbb{R}^6$ and $\Xi$ is the unit simplex. Note that $S(p,p)=1$ and $S(r,s)=0$ for $(r,s)=\operatorname{argmax}_{p,q\in S}d(p,q)$. Since $\hat{p}_{k,n}\to p_{k}$ a.s., the sample version $\hat{S}:=S(\hat{p}_{1,n_1},\hat{p}_{2,n_2})$ converges a.s. to $S(p_1,p_2)$.

---

Since $\hat{S}$ is random, obtaining a particular number doesn't provide much information (even if the true parameters are the same, a particular realization of $\hat{S}$ can be close to $0$). A statistical way to assess similarity between two distributions would be testing the following hypothesis: $$ H_0:p_1=p_2, \\ H_1:p_1\ne p_2. $$

First, by the CLT, $$ \sqrt{n}\left(\hat{q}_{k,n}-q_k\right)\xrightarrow{d}N(0,V_k), $$ where $q_k= p_{k,1:5}$, $\hat{q}_{k,n}=\hat{p}_{k,n,1:5}$, and $V_k=\operatorname{diag}(q_k)-q_k q_k^{\top}$.

Assume that the sample sizes are $n_l$ and $m_l$ such that $n_l,m_l\to \infty$ and $m_l/n_l\to 1$ as $l\to\infty$, and let $r_l=(n_l+m_l) / 2$. Since $\hat{p}_{1,n_l}$ and $\hat{p}_{2,m_l}$ are independent, $$ \sqrt{r_l}\left(\hat{q}_{1,n_l}-q_1\right)-\sqrt{r_l}\left(\hat{q}_{2,m_l}-q_2\right)\xrightarrow{d} N(0,V_1+V_2). $$

Therefore, one may consider the following test statistic: $$ T_l:=r_l(\hat{q}_{1,n_l}-\hat{q}_{2,m_l})^{\top}(V_1+V_2)^{-1}(\hat{q}_{1,n_l}-\hat{q}_{2,m_l}). $$ Under $H_0$, $V_1=V_2$ and $T_l\xrightarrow{d}\chi_5^2$ (in practice, $V_k$ is replaced by any consistent estimator). Thus, one rejects $H_0$ when $T_l>\chi_{5,1-\alpha}^2$, where $\chi_{5,1-\alpha}^2$ is the $(1-\alpha)$-quantile of $\chi_5^2$.

Answer 2

I am thinking of two ways to approach the question. You could go down the path of considering rolling both at the same time and counting when the values match. I believe your PDF would be standard, with a center of 1/36. This would be a kind of binomial test with success meaning a match and a failure being not a match. You would want a one-tailed test to measure bias on the upper side (frequency above 1/36).

Alternatively, there is such a thing as a multinomial test, which would look at probabilities of die rolls for each dice. I believe this would be more difficult, unless you are concerned with specific values on each dice. From the way I interpret your question, it seems that you just care about the times that the dice output the same value (perhaps to catch two dice that add to 7 or 11).