Cautions For Confidence Intervals and Significance Tests

-When using data from a sample to perform a significance test or create a confidence interval, the data must be from an SRS, or from a sample chosen close enough to randomly that we can regard it as an SRS.

-x is strongly influenced by outliers. The sample should be examined beforehand to determine if any outliers are present, and if so, if they can be corrected or removed.

-Small samples must have roughly normal distributions for the sampling distribution of x to be normal, so confidence intervals and significance tests wont be accurate for small non-normal samples.

-Multiple analysis greatly increase the chance of an error. For example, if we create 20 90% confidence intervals for a parameter from 20 samples, the probability is quite high that at least one of our confidence intervals will fail to capture that parameter.

-The margin of error in confidence intervals and the level of significance of significance tests do not take into account any errors in the data itself. (e.g. bias, undercoverage, nonresponse etc.)