Probstat/notes/hypothesis testing

This is part of probstat.

Since it is very hard to obtain complete information of the population, we usually end up with a collection of much smaller sample data. A question arises: how can we be confident if the conclusion we make from the collected sample is correct or it is only by chance?

This section tries to answer this question.

A motivating example

You friend gives you a coin and claims that this coin is special. (It is unclear what special is about this coin.)

You want to prove it so you toss the coin for 20 times.

If you get 10 heads, do you believe your friend that the coin is special?

If you get 12 heads, do you believe your friend that the coin is special?

How about 15 heads? How about 18 heads? How about 20 heads?

Let's consider each case.

10: What is the probability that a normal coin turns up at least 10 heads from 20 coin tosses? 58% So that this does not show anything special about this coin.

12: What is the probability that a normal coin turns up at least 12 heads from 20 coin tosses? 25% So this coin might be a bit special?

15: What is the probability that a normal coin turns up at least 15 heads from 20 coin tosses? 2% This coin is special or I am very lucky.

18: What is the probability that a normal coin turns up at least 18 heads from 20 coin tosses? 0.02% This coin is special or I am extremely lucky.

20: What is the probability that a normal coin turns up at least 20 heads from 20 coin tosses? about 1 in a million. I definitely should believe that this coin is special.

Let's go back to our reasoning in the previous coin example.

We want to reject some belief, i.e., that the coin is normal. In this case, that the normality is in the probability of turning up head. So the hypothesis that we want to test (or reject) is the following:

H₀: "the probability that the coin turns up head is 0.5".

If the experimental result contradicts this hypothesis, we can reject it. However, note that it is impossible to completely contradict this hypothesis, even with a result that shows 1000 heads in 1000 coin tosses does not contradict this hypothesis because there is non-zero probability to obtain that result. Therefore, we are happy with a result which is "unlikely" enough. The degree of "unlikely" matters in our confidence in rejecting the hypothesis.

Consider this criteria:

We shall reject $H_{0}$, if after tossing the coin for 20 times, we get at least 18 heads.

We know that if the hypothesis $H_{0}$ is true, the probability that we reject it is at most 0.02%. Therefore, if we reject it under this assumption, it is extremely unlikely because of chance. The probability that we reject $H_{0}$ when it is actually true is the significant level of the test; in this case, the level of significant of the test is $\alpha =0.0002$. (Note that if the significant level is very small, it means that if we reject $H_{0}$, it is very significant.)

The null hypothesis

See also the wikipedia article.

When we perform hypothesis testing, we usually start with a hypothesis that describe a "normal" situation, usually referred to as the null hypothesis. This hypothesis is there so that we can accept or reject it with experimental data.

In the previous example, the null hypothesis specifies that the head probability of the coin is 1/2. Let's consider another example. Suppose that we have know that on average students will get 80 points from the final exam for the probability class. In this semester, we try something different. We add another review section for each week and we would like to test if this review section improves the test score. We let the students with review section take the final exam let $\mu$ denote the average score. Our null hypothesis is

H₀: $\mu \leq 80$.

After we set up the null hypothesis, we should create a criteria for accept or reject the null hypothesis.