Statistical significance in the interval

27 Views Asked by At

The statistical significance of an observed result is not always the same as the material significance - the practical importance - of that result.

I am curious as to why this would be the case in some circumstances. And is there an example when a statistically significant result is also materially significant, and vice versa?

1

There are 1 best solutions below

0
On

Statistically significant, not of practical importance. I once consulted on a study to determine whether a new machine and a standard lab test gave the same results for hemoglobin (Hgb) in the blood. Hgb was determined for about three dozen newborn babies using both methods for each. The purpose of the study was to see if the new machine (quicker, easier) could be used instead of the lab method.

The machine gave slightly higher values for most of the babies, to the extent that a paired t test indicated a highly significant difference. But that did not mean that the machine couldn't be used. (a) The difference, although almost certainly 'real', was so small as not to be clinically important. (b) A slight adjustment to the machine could make it give values that matched the lab results. (No one knows or cares which was really more accurate.)

Statistically significant, and of practical importance. Often, with some prior knowledge of the variances involved, it is possible to design a study so that statistical significance matches practical importance. This is done by specifying the size of the difference that is of practical importance, and making sure that the sample size is just right that the power of the test for that difference is reasonably high (say 95%). Thus the study is 'tuned' in advance so statistical significance and practical importance are quite likely to coincide.