Probability on a null hypotheses?

Question

I found this question from Ian Hacking's book on probability. I understand that this question probablly uses standard deviation due to the 0.01, but what doesn't make sense is what exactly is a null hypotheses, and how would I determine if the study is significant or not?

---

Suppose the government is testing a new education policy. There are only two possibilities: either the policy will work and it will help high school students learn to write better 3/4 of the time, or it will have no effect and students’ writing will only improve 1/2 of the time, as usual.

H : 3/4 probability of improvement for each student.

¬H : 1/2 probability of improvement for each student.

The government does a study of 432 students and finds that, under the new policy, 285 of them improved in their writing.

(a) If the null hypothesis is ¬H, are the results of the study significant at the .01 level?

(b) If the null hypothesis is H, are the results of the study significant at the .01 level?

Answer 1

This is an unrealistic situation. Why would one suppose it is only possible for the new government policy to result is 3/4 of the students improving or to result in 1/2 improving? And nothing in between?

In fact, the data suggest that about 66% have improved. So it seems we have a government policy, which (as for so very many government policies before it) has delivered considerably less than promised.

@BrianTung has made a valiant effort (+1) to suggest options. I will just crunch some numbers along lines of the instructions as I understand them.

First, whatever hypothesis you test, you have the estimate $\hat p = 285/432 = 0.6597$ of the true improving proportion $p.$

Suppose we test $H_0: p = 3/4.$ Then the z-statistic is $$Z = \frac{\hat p - .75}{\sqrt{\frac{.75(.25)}{432}}} = -4.34.$$ Against a two-sided alternative, you would reject at the 1% level if $|Z|>2.576$, which is true. Against a left-sided alternative, you would reject at the 1% level if $Z < -2.236,$ which is also true. So you would reject $H_0$ whether the alternative was intended to be left- or two-sided. (It is late here, I should have gone to bed an hour ago, and you had better check all of my numbers.)

Similarly, testing $H_0: p = 1/2,$ the z statistic is $$Z = \frac{\hat p - .50}{\sqrt{\frac{.50(.50)}{432}}}.$$ Again I get an answer that leads to rejection at the 1% level against either a right- or two-sided alternative.

Thus, all reasonable scenarios seem to lead to rejecting both hypothesis. That, in spite of the fact that the question seems to me to lead in the direction of saying one of them must be true.

Note: I could have down-voted your question as nonsense, and voted to 'close' it in the grounds that it is 'unclear what you are asking'. But it seems you have cogently figured out that the problem makes no sense, and you are wondering if you have missed something. Maybe so, but I can't make sense of it either--in case that is any consolation.

Answer 2

Basic approach. The null hypothesis is the thing you try to reject. In particular, we suppose the null hypothesis were true: Then what would be the probability of obtaining results like the ones we obtained? If the probability is lower than a given value of $p$ (such as $p = 0.01$), then we consider the results significant and we reject the null hypothesis; otherwise, the results are insignificant. It is useful to note that insignificant results do not prove the null hypothesis; they only fail to reject it.

Usually, the null hypothesis is some notion that a test variable has no impact on results. In this case, your problem has two different hypotheses, and they're asking you to treat each one, separately, as the null hypothesis, and see which (if either) is rejected.

Some importance is placed on the meaning of "like the ones we obtained". For one-dimensional variables, sometimes we are interested in a one-sided test, and sometimes we are interested in a two-sided test. For instance, suppose the null hypothesis is "Eating carrots has no impact on IQ test performance." Then we would be interested in a two-sided test, since both decreased results and increased results would be equally significant. But if the null hypothesis were "Eating carrots has no beneficial impact on IQ test performance", then we would want a one-sided test, since we don't care if the test performance goes down.

You have to use your judgment to decide whether which kind of test you should be doing, and then what the proper distribution (often, but not always, a Gaussian) you should use to evaluate the test.