The Difference Between Statistical Significance and Practical Significance

You just ran your analysis and the p-value is 0.03. You're thrilled — your result is statistically significant! But before you start celebrating, your advisor asks: "Is it practically significant?"

This question has derailed many a dissertation defense. Let's make sure it doesn't derail yours.

Statistical Significance: Did Something Happen?

Statistical significance means the result you observed is unlikely to have occurred by chance alone. When p < 0.05, you're saying: "If there were truly no effect, there's less than a 5% probability I'd see a result this extreme."

That's it. That's all a p-value tells you. It says nothing about whether the effect is big, important, or useful.

Practical Significance: Does It Matter?

Practical significance asks whether the effect is large enough to be meaningful in the real world. A tutoring program that raises test scores by 0.5 points on a 100-point exam might be statistically significant with a large enough sample — but no school administrator would restructure their budget for half a point.

Practical significance is measured through effect sizes — standardized metrics like Cohen's d, eta-squared, or R-squared that quantify the magnitude of your finding.

Why Large Samples Create Confusion

Here's the trap: with a large enough sample size, any difference becomes statistically significant — even a trivially small one. If you survey 10,000 people, a 0.2-point difference on a 50-point scale might yield p < 0.001. That result is highly significant statistically but completely meaningless practically.

This is one of the most important concepts in applied statistics, and it's the reason your committee asks about both types of significance.

A Real-World Example

Imagine you're studying whether a new teaching method improves reading comprehension. Your results:

• Control group mean: 74.2
• Treatment group mean: 75.1
• p-value: 0.02
• Cohen's d: 0.08

The p-value says the difference is statistically significant. But Cohen's d of 0.08 is tiny — far below the benchmark of 0.2 for a "small" effect. The teaching method produces a real but negligible improvement. Is it worth the cost and effort of implementation? Probably not.

Now imagine different results:

• Control group mean: 74.2
• Treatment group mean: 82.6
• p-value: 0.01
• Cohen's d: 0.75

Here you have both statistical significance (p < .05) and practical significance (a medium-to-large effect). This is the kind of result that changes practice.

How to Address Both in Your Dissertation

1. Report p-values for every inferential test — this is standard practice.
2. Report effect sizes alongside every p-value. APA style requires this.
3. Interpret effect sizes in context. Don't just say "medium effect." Explain what that means for your specific population and setting.
4. Discuss practical implications in your Discussion chapter. Even if a result is statistically significant, be honest about whether the effect is large enough to matter.
5. Use confidence intervals to show the range of plausible effect sizes.

The Bottom Line

Statistical significance answers: "Is there an effect?" Practical significance answers: "Is the effect big enough to care about?" Your dissertation needs to address both. The easiest way to do that is to always report and interpret effect sizes — not just p-values.