I am currently studying hypothesis testing for dependent two-sample (proportion). The crux for my question is this, what test does one use to compare the proportion of multiple samples for non-dichotomous data (i.e., multiple categories)? For example, a 1 to 5 star rating by 50 critics for 5 restaurants.
The context in which my question arises from is as follows. For comparing the sample proportions ($H_0:\pi_1=\pi_2$) of dependent two-samples for dichotomous data, we use the McNemar test. Going with an example, let us say we are comparing the effectiveness of 2 drugs, drug (1) and drug (2). We would employ the McNemar test if we have data like this,
| Effective | Not effective | |
|---|---|---|
| Effective | $n_{11}$ | $n_{12}$ |
| Not effective | $n_{21}$ | $n_{22}$ |
Continuing with this example, if we wanted to compare the effectiveness of these 2 drugs on multiple categories (i.e., non-dichotomous), we cannot use the McNemar test. Instead, we would employ the Bhapkar test or Stuart-Maxwell test for data like this,
| Effective | Somewhat effective | Not effective | |
|---|---|---|---|
| Effective | $n_{11}$ | $n_{12}$ | $n_{13}$ |
| Somewhat effective | $n_{21}$ | $n_{22}$ | $n_{23}$ |
| Not effective | $n_{31}$ | $n_{32}$ | $n_{33}$ |
If we had wanted to test for multiple samples (> 2 drugs; $H_0:\pi_1=\pi_2=\pi_3$...) on dichotomous data, we also cannot employ the McNemar test. Instead, we would employ the Cochran Q test for data like this,
| drug (1) | drug (2) | drug (3) | |
|---|---|---|---|
| person (1) | $0/1$ | $0/1$ | $0/1$ |
| person (2) | $0/1$ | $0/1$ | $0/1$ |
| person (3) | $0/1$ | $0/1$ | $0/1$ |
| person (4) | $0/1$ | $0/1$ | $0/1$ |
| person (5) | $0/1$ | $0/1$ | $0/1$ |
Where,
- 0 = Effective
- 1 = Not effective
To summarise the properties of the 3 tests,
| No. of samples | No. of categories | |
|---|---|---|
| McNemar | 2 | 2 |
| Bhapkar / Stuart-Maxwell | 2 | > 2 |
| Cochran's Q | > 2 | 2 |
| ? | > 2 | > 2 |
This goes back to my question, what test can we use for comparing the proportion of multiple samples with multiple categories?
p.s. I know that we can employ Cohen/Fleiss Kappa to analyse "agreement" for multiple samples/categories. What I am asking is for hypothesis testing of sample proportion equality (i.e., $H_0:\pi_1=\pi_2=\pi_3$...).
Please correct me if my understanding of the subject matter above is wrong. I am very much in the process of trying to grasp this subject and greatly appreciate any further insight on the matter. What I have stated in this post is based on my personal reading of the subject thus far.