Posts

Showing posts from February, 2025

A/B Testing, True Knowledge, and Power Analysis

Image
Over the years, I've run into a few scenarios that piqued my interest in experiment design. To be honest, when some of my stats friends were taking classes on experimental design in college, I thought those classes sounded boring as hell. Something about about how to frame survey questions, or maybe calculating margin-of-error for public opinion polls. In practice, understanding the limitations of experiments is extremely useful for both  planning  and interpreting  A/B tests! Stat-Sig A/B Test != True Knowledge A/B testing is widely used at tech companies and beyond. Meta is no exception. Even though it's so widely used, many, many people don't always fully understand it's nuances; and occasionally that causes issues when people see confusing results and want an explanation. In the sciences, similar issues have surfaced with regard to the "reproducibility crisis."  To start off, the hardest lesson I've learned about statistics is that we don't really ...

🚀 And we're off!

For the past 5 years, I've worked at Meta in Software Engineering and Machine Learning roles, spending most of my time working on Smart Glasses. This month, I was fired as one of ~5,000 employees openly referred to as "under-performers." I vehemently reject that characterization. In April of last year, I had talked to the Smart Glasses team about rejoining their team as a Machine Learning Engineer. Both then and now, I was fully transparent: I wasn't sure that the role was the right fit. I had been working on ML for Dialog for years, and I had been working in the Ads space for a year as a Machine Learning Engineer, but my time as an ML Engineer focused on graph embeddings and ranking models. I was familiar with LLMs and Speech modeling at a high level, but I wasn't an expert. My director told me not to worry about not being a Voice AI modeling expert. He said something to the effect of, "Who would have more context than you?"  Ultimately, I decided to ta...