What is the difference between ANOVA and ANCOVA?

Question

In the context of using only experiment data for ANOVA analysis, ANCOVA offers post hoc statistical control. Is this a valid conclusion and why?

Answer 1

Usually 'post hoc' means making inferences inspired by what you see in the data. This is always dangerous because even random data may appear to have interesting patterns. I am not saying that ANCOVA is never used for 'post hoc' analysis, nor that some kinds of 'post hoc' analysis are not intended for 'statistical control'. But without knowing more, and admittedly without context, I am instantly skeptical of what may lie behind your statement.

So let me try to describe the actual difference between ANOVA and ANCOVA. Suppose you are testing five new kinds of keyboards, each designed to mitigate the kind of repetitive stress injury that leads to carpal tunnel disease. Subjects are office workers who have complained of such repetitive stress. Each worker is evaluated before and after a two-week period of time using one of the new keyboards and the change in degree of pain is described numerically for each subject. This is the response variable for the experiment.

A one-way ANOVA would have five groups, one for each type of keyboard under test. Suppose we have 15 subjects per group. Without 'controlling for' other factors as a pre-planned of the experiment, our experiment will likely be a disaster.

A couple of possible difficulties may be: (1) Some people have jobs that require more keyboarding that others. (2) Some people will adapt readily to a change in keyboard and some will take more than the two-week experimental period to adapt. One artifact might be that the worst keyboard appears to be best. Subjects will hate using it and put off heavy keyboarding until after the experiment is over, so they will appear to show magnificent decrease in pain from assignment to the horrible keyboard.

If we have advance information about one of these factors, we might use it as part of the ANOVA design. For example, we might have type of keyboard as one factor, and whether the job requires Heavy, Medium, or Light keyboarding as a second factor. Then we would have a two-factor ANOVA with $5 \times 3$ 'cells' in the design. If we plan 15 subjects per keyboard type, perhaps we can obtain subjects so that we have 5 per cell. In this design subjects would have to promise to do their normal amounts of keyboarding during the period of the study.

However, it might be difficult to find $5 \times 3 \times 5$ subjects to assign to the 15 cells. And it might be awkward to get subjects to comply with their promise to do their normal amounts of keyboarding. Another approach would be to collect information on actual hours $H$ of keyboarding, and treat it as a covariate. Perhaps also to ask subjects how easily they adapted to the new keyboard of a scale $A$ of 1 to 10. Then we could treat $H$ and $A$ as covariates. Many ANCOVA designs and analytic paths are possible. The design would be certainly be chosen in advance of collecting data. Ideally, the analytic paths would also be specified in advance by the protocol for the experiment.

In any experiment, it is possible that, upon seeing the data, additional tests and methods of analysis come to mind. Strictly speaking, any findings from these 'post hoc' procedures would not be reported with the same level of assurance as the analyses from the original design and protocol. They should go into the 'possible future work' section of the report, and might be pursued in future studies when they are pre-planned features of a new project.

Reporting 'ad hoc' results on the same level as pre-designed ones is the essence of 'irreproducible results' so much in the news lately.

Acknowledgment: An experiment somewhat like the one I describe here is reported and analyzed in Chapter 17 of Gary Oehlert's book on the design of experiments, available here. You can read a very nice formal description of ANCOVA there.