Why does ANOVA (and related modeling) exist as a separate technique when we have regression?

Question

I've asked this question to a number of statistics PhDs and the like, but have never gotten close to a satisfactory answer. I understand that ANOVA forms the basis for the hypothesis test(s) underlying regression, and that, in a sense, the two techniques are two sides of the same mathematical coin. However, it seems to me that regression gives you everything you would want from ANOVA (i.e., tests of significance), and more (e.g., the ability to use quantitative independent variables and estimates of effect sizes), so I don't understand why ANOVA exists as a standalone method/framework/practice, as opposed to simply being a method for testing regression.

Answer 1

Why in ML literature coefficients are called "weights" and explanatory variables "features"? I haven't asked CS/ML PhDs "and the like", but I guess that it stems from different background and motivation that led to the development and use of these tools. My guess is that ANOVA point of view is much more suitable for experiment design than the regression point of view. A lot of literature in the field of ANOVA deal with balanced and unbalanced designs and how it effects the inference on main or non-main effects.

The "regular" regression point of view is more common in econometrics where there are no control experiments and there is no meaning to talk about balanced and unbalanced cells as no "design" is possible. As such, there is no much use of the endless discussion about various types of sums of squares to interpret the "marginal" effect of some factor. IMHO, these peculiarities of ANOVA concerned mainly with interpretation of the model rather than with its intrinsic mathematical features. Moreover, it is pretty uncommon to encounter economic model with only categorical explanatory variables. To sum it up, the classical regression point of view is more "mathematical" and concerned much more with statistical and probabilistic features of the model as an abstract description or approximation of some natural phenomenon. While the more social-science prone ANOVA point of view is concerned much more with interpretation and design of the experiments to fit these kind of analyses. It is pretty uncommon to talk about ANOVA or even ANCOVA as approximation of some "data generating model", but rather just as statistical tool for post-hoc analysis.

I think that similar relationship you can find in the use of regression and specific of logistic regression in classical social science analysis compared with the ML approach. IMHO, ML literature jargon calls it supervised learning and concerns mainly with its efficacy as a classifier, while in economics and psychology its main use is for probability and odds-ratio estimation.