At a recent statistics workshop I was conducting to a group of medical students I asked how many could define statistically significance. Everyone had heard of it and knew that p < .05 constituted a statistically significant effect, but no one really understood what it meant.
To understand what it means, it is crucial to remember that statistics is typically used for 2 purposes: 1) to describe and 2) to make inferences about a population from a sample drawn from the population.
With few exceptions, when we conduct a study, the data we collect is from a sample. But it is not really the sample per se in which we are interested. We almost always want to be able to generalize from the sample to the population from which the sample was drawn. If we administer a new painkiller to one group of individuals and a placebo to another group, and find that the results indeed show greater pain reduction in the drug group, we need to know whether the difference obtained from the sample reflects a “real” difference in the population or whether, alternatively, there is no difference between the groups and the observed difference is due to chance alone.
Let’s look at an example.Continue reading “Statistical significance: What it does and does not mean”