Significance Tests for Population Means

Significance tests are a type of statistical inference where information from a sample of a population is used to assess the validity of a claim made about that population. Significance tests look at the properties of the sample, and work out the probability of obtaining such a sample if a given claim about the population is true. This probability is called the "P-value" of the test.

Significance tests for a population mean(µ) are performed according to the following procedure:

1. Null Hypothesis
We state the claim we are testing. This claim is called the null hypothesis and is represented by H0.
e.g. I believe that on average I stay awake for the first 5 minutes of my lectures (µ = 5). This null hypothesis is written as:
H0: µ = 5

2. Alternative Hypothesis
We state an alternative hypothesis (Ha) which we suspect is the case.
e.g. Ha: µ ≠ 5
That is, I don't stay awake for 5 minutes on average.
Hypothesis can be two-sided as in the example above, or one-sided
e.g. Ha: µ > 5
That is, I stay awake for more than five minutes on average.
Or Ha: µ < 5 (I stay awake for more than five minutes on average.)

3. Sample
Obtain a sample of size n from a population. Calculate the mean of the sample (x) and its standard deviation (s)

4. Standardise x
If we know σ
We know that the sampling distribution of x has mean µ, and if we assume H0 is true we know that µ = 5, so we have enough information to standardise x. This tells us how far x is from µ in units of standard deviations assuming H0 is true. We can then determine from a table of standard normal probabilities, the probability of obtaining such an x if H0 is true. If the probability is very low, our assumption that H0 is true is probably wrong.

To standardise x we use the formula:

where µ0 is the mean assuming H0 is true.
z is called the "one-sample z statistic".

If we don't know σ
Usually we don't know σ, so we must instead use s as an estimate. When we standardise x using s we get a t-distribution with n-1 degrees of freedom, just as with confidence intervals. The probability of obtaining such a t-value can be found using software or a table or compared to a table of t critical values.

To standardise x here we use the formula:

t is called the "one-sample t statistic"

The one-sample z and t statistics fall under the general category "test statistics."
I'll ignore z for the rest of this post as in reality we never know σ.

5. P-Value
The probability of getting a result at least as far from µ as t assuming H0 is true is called the P-value, or just P.
P is also the probability that H0 is true.
P can be calculated exactly using software, or more roughly using a table of t critical values.
To find p using tables of t critical values we find |t| (the absolute value of t) and compare it to the t critical values (the t values for certain probabilities) in the row df = n - 1. If n - 1 is different from the dfs in the table, take the next df below n-1.
If Ha is two-sided we now double P.
e.g. if 0.01<P<0.02 when Ha is one sided, then 0.02<P<0.04 when Ha is two-sided.

Table of t critical values (click to enlarge)

(The P-values are the numbers at the top of each column)

The results of significance tests can be described as significant or not significant at certain levels. The result of a test is significant at level α if P < α.
e.g. the results of a test is significant at level 0.05 if P < 0.05
The smaller P is, the more significant the results, and the more evidence we have against H0.

As a rough guide for when it's safe to use this t-procedure, use the guidelines for one-sample t-procedures