A/B Testing, True Knowledge, and Power Analysis
Over the years, I've run into a few scenarios that piqued my interest in experiment design. To be honest, when some of my stats friends were taking classes on experimental design in college, I thought those classes sounded boring as hell. Something about about how to frame survey questions, or maybe calculating margin-of-error for public opinion polls. In practice, understanding the limitations of experiments is extremely useful for both planning and interpreting A/B tests! Stat-Sig A/B Test != True Knowledge A/B testing is widely used at tech companies and beyond. Meta is no exception. Even though it's so widely used, many, many people don't always fully understand it's nuances; and occasionally that causes issues when people see confusing results and want an explanation. In the sciences, similar issues have surfaced with regard to the "reproducibility crisis." To start off, the hardest lesson I've learned about statistics is that we don't really ...