Can I accept the null hypothesis?

Question

I'm running an ANOVA on some data that I have. I get that if my p-value is less than 0.05 (with a significance of 5%) I can reject the null hypothesis and accept that the means between my groups are different, but that doesn't seem to be the case in my data, in fact, the p-value is greater than 0.95. Does this mean that I can, with statistical significance, accept the hypothesis that the means of my groups are equal?

Answer 1

Not really. Basically the alternative hypothesis is what you want to prove. Thus you assume the Null hypothesis unless there is a significant reason to believe otherwise.

For this one intruduces the error of first order and the error of second order. The error of first order is basically: How big is the probability that if the Nullhypothesis is in fact true we are rejecting it. And the error of second order is: How big is the probability that if the Nullhypothis is not true we are not rejecting it.

One then constructs a test in such a way that the error of first order is lower than some threshold.

Now, the $p$-value is basically: What is the largest threshold so that I would just reject the nullhypothesis. So if your $p$-value is $0.95$ that if the Nullhypothesis is true you’d expect to get something as extreme as your result in 95% of the cases.

But if you want to turn around the whole thing you’d need to make the error of second order small.

EDIT: You might want to look into the so called power of a test, which is the counterprobability to the error of second order. Basically the power says: If the alternative hypothesis is true, how likely is it that we’ll determine this correctly to be.

Even if the $p$-value is high that does not mean that the power is high.