Survey vs interview size

80 Views Asked by At

I have a survey of size 120 and 20 interviews. Can I compare the results? Both groups answered same questions.

Example: let's say a question was do you like to read novels ? Can I say that 20% of those interviewed said yes while 60% of survey said no? I mean are they comparable? Can I say there was an association between age and reading novels of the survey group, but not the interview group? Won't there be a difference in terms of the size needed of 2 groups? Is 20 enough?

Thanks !

1

There are 1 best solutions below

2
On

Do you mean you have 120 self-enumerated surveys (paper or web) and 20 interviewer-enumerated surveys (face-to-face or phone)? If so, then the answer is "Yes but". What matters isn't so much the number of each - although 20 is a bit of a small sample for doing much - but the modal bias, the tendency for people to respond to the same questions differently when they are presented differently.

If you're trying to see whether this bias exists, then you can certainly compare the results of the two and discuss that effect, but with the small sample sizes you probably won't measure much.

On the other hand, if you're hoping to combine the two samples to give you a total sample size of 140, then you probably want to conduct some tests against the hypothesis that the two come from a common distribution, such as the Kolmogorov-Smirnov test for continuous variables, and probably just a straight binomial or chi-squared test for categorical variables. If you cannot reject the null hypothesis (that the two samples come from the same distribution), then you should be safe combining the two and drawing inference from the sample of 140.

Edit: Based on your new edit, the answer is yes you can make those kinds of comparison, although the small sample size (especially for the interviews) will hamper you a lot in saying that any differences you see are meaningful - with only 20 people, the standard errors on things like proportions are going to be a bit on the large side, and I wouldn't be trying to look too hard at any correlations or associations, because a single outlier respondent will throw pretty much everything off.

The other thing is, what is your research question - what are you trying to find out? If your question is "Is there a difference between the interview group and the self-enumerated survey group?", then go ahead and do hypothesis tests about differences in means or whatever you want. If your question is "What proportion of people like to read books?" then first try a test like I describe above, then if the test doesn't suggest there are major differences you can combine the samples to calculate the estimate.

If you're worried about the fact that there is a large difference between the sample sizes, that isn't really a factor in saying whether you can compare them - what's more important is how they were selected, whether they represent different populations, and whether there's any sign of modal bias.