I have different machine learning models and losses, each trained using 5-fold cross-validation. Would it make sense to run a two-way ANOVA two evaluate which are statistically performing better? Example:
| Losses | Model 1 | Model 2 | Model 3 |
|---|---|---|---|
| Loss 1 | Accuracy Fold 1, ..., Accuracy Fold 5 | ... | ... |
| Loss 2 | ... | ... | ... |
| Loss 3 | ... | ... | ... |
Would I break any assumptions by having the accuracies on different test sets (cross-validation by definition)?