In this lesson, we'll learn about hypothesis testing and statistical significance. A hypothesis is a pattern or rule about a process that can be tested, and hypothesis testing helps us determine if a change we made had a meaningful impact or not.

For example, hypothesis testing can help you determine:

  • If a new banner ad on a website caused a meaningful drop in user engagement.
  • If raising the price of a product caused a meaningful drop in sales, or
  • If a new weight loss pill helped people lose more weight.

Just because user engagement or sales decreased after adding a new banner doesn’t necessarily mean the banner caused the decline -- it could have been random chance. Every process has an inherent amount of randomness, and hypothesis testing helps us better understand the role of chance so we can improve our conclusions.

As with all our courses, you can apply what you’ve learned in our in-browser exercises, and automatic answer checking will help you know whether you’ve mastered each concept.

Objectives

  • Learn how hypothesis testing works.
  • Learn how statistical significance and hypothesis testing are related.

Lesson Outline

1. Hypothesis testing
2. Research design
3. Statistical significance
4. Test statistic
5. Permutation test
6. Sampling distribution
7. Dictionary representation of a distribution
8. P value
9. Caveats
10. Next steps
11. Takeaways