Finding correct statistical test for my study

Question

So I have a study that I have carried out with 40 test subjects. The study consisted of reading two different texts using different methods and answering a number of questions corresponding to each text. Half the population read text $t_1$ using method $m_1$ and $t_2$ using method $m_2$. The other half read text $t_1$ using method $m_2$ and text $t_2$ using method $m_1$.

I'm trying to figure out if there's any inherent difference in difficulty between the two texts, but I can't figure out which statistical test to use. It feels as if a paired t-test is wrong since the same person read the two texts using different methods, but I'm not really sure what else to use.

Answer 1

IMO you since are doing comparisons between four different groups (results sets), an ANOVA (Analysis of Variance) approach would be the best approach.

1. Analyse and look at your data, and remove any outliers first.

2. Check if the all the assumptions for ANOVA have been satisfied.

I will presume that the assumption of the dependent variable being normally distributed for each group is not satisfied (can be checked via anderson-darling test, and graphs of the data).

3. Now you need to check for homogeneity of variances. One can do this via Levene's test.

$H_0: \sigma_1 = \sigma_2 = \sigma_3 = \sigma_4$ $H_1: \sigma_i \ne \sigma_j , i\ne j, i,j=(1,2,3,4)$

https://en.m.wikipedia.org/wiki/Levene%27s_test

After obtaining the results of Levene's test:

i. If any of the p-values are greater than or equal to $0.05$ then:

1. $H_0$ is rejected and homogeneity of variances assumption is violated, as well as normality.

2. This would imply a nonparametric ANOVA is appropriate - such as Kruskal-Wallace. Software such as SAS, SPSS have this. MS Excel may have also.

   https://en.m.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance

   Read section "Reasons to Use Parametric Tests" on http://blog.minitab.com/blog/adventures-in-statistics-2/choosing-between-a-nonparametric-test-and-a-parametric-test

   If there are significant differences found between the 4 groups, you will need to do post adhoc comparisons for determining which of the four groups are significantly different from each other. Holm's t-test could be used for these.

ii. Otherwise if all p-values are less than $0.05$ then one can assume that $H_0$ is accepted - all variances of groups are the same. Then you would need to research which test to do next (with non-normal data, and equal variances).