What Is Statistical Significance?

Statistical significance is a way of judging whether a result in a study is likely to reflect something real, or whether it could easily be explained by random chance. Researchers typically test a null hypothesis, which assumes that there is no effect or difference, and calculate a p-value that says how surprising the observed results would be if the null hypothesis were true.

The p-value

A p-value is the probability of obtaining results at least as extreme as those observed, assuming that the null hypothesis is true. Researchers choose a threshold before the study, called the significance level or alpha, often 0.05. If the p-value falls below it, the result is described as statistically significant, and the null hypothesis is rejected.

Example. A study compares test scores of two groups, and finds a difference with p = 0.03. This suggests that a difference this large would be unusual if the teaching methods were equally effective. It does not prove that the new method works.

What it does not mean

• It is not the probability that the hypothesis is true.
• It does not measure the size of the effect. Very large studies can find tiny, trivial effects that are significant.
• It does not guarantee replication.
• It does not prove cause, which depends on the design.
• A p-value just below 0.05 is not much different from one just above it.

Effect size and confidence intervals

Effect sizes describe how large a difference or relationship is, and confidence intervals give a range of plausible values. Together with p-values, they give a fuller picture. Many journals now ask authors to report them.

Common problems

• P-hacking: trying many different tests and reporting those that cross the threshold.
• Multiple comparisons: running many tests increases the chance of false positives unless corrected.
• Small samples: can lead to unreliable results.
• Publication bias: significant results are more likely to be published.
• Misinterpretation, such as treating non-significance as evidence of no effect.

Reading research as a student

When you see "significant," ask by how much, for whom, in what context and how big the effect is. Look for effect sizes, sample sizes and how the study was designed. Be skeptical of headlines that say "proven" based on a single p-value.

Why it matters

Statistical significance is central to many fields, and understanding it helps you interpret results, write about them and avoid common errors.

Related terms

See variable, correlation vs causation, peer review, IRB and dissertation defense.