keita-dc / dsc-statistical-power-anova-introduction-dc-ds-060319

Statistical Power and ANOVA - Introduction

Introduction

In this section you'll continue to deepen your knowledge of hypothesis testing and t-tests by examining the concept of power; an idea closely related to type II errors. With that, you'll see how the rate of type I errors, power, sample size and effect size are intrinsically related to one another. From there, you'll investigate issues surrounding running multiple A/B testing experiments, which will lead you naturally into the concept of ANOVA - Analysis of Variance which allows you to effectively conduct multiple A/B tests simultaneously, testing for the influence of multiple factors all at once.

Objectives

You will be able to:

• Define statistical power
• Define ANOVA
• Understand where ANOVA is used

Statistical Power

Statistical power is equal to $1 - \beta$ where $\beta$ is the rate of type II errors. As you will see, power is related to $\alpha$, sample size and effect size. Typically a researcher will select an acceptable alpha value and then examine required sample sizes to achieve a desired power such as .8 (or higher).

Welch's t-Test

After an initial exploration of statistical power, you'll take a look at Welch's t-test. This is an adaptation of the unpaired student's t-test you've seen previously which allows for different sample sizes or different variances between the two groups.

Multiple Comparisons

From there, you'll look at some of the issues that arise when trying to perform multiple comparisons - from the risks of spurious correlations to the importance of corrections such as the Bonferroni Correction to deal with the cumulative risks of type I errors inherent in multiple comparisons.

ANOVA Testing

Finally, you'll take a look at the more generalized procedure for conducting multiple comparisons: Analysis of Variance or ANOVA. You'll see that ANOVA of only two groups is statistically equivalent to a two sided t-test. That said, ANOVA fully supports comparing multiple factors simultaneously.

Summary

Without a good understanding of experimental design, it's easy to end up drawing false conclusions. In this section you'll cover a range of tools and techniques to deepen your understanding of hypothesis testing and ensure that you design experiments rigorously and interpret them thoughtfully.