Experimentation & A/B Testing

I Ran Billions of Simulations to Simplify Multi-Armed Bandit Algorithms for You

Whenever anyone mentions Multi-armed bandit algorithms, the synonymous term reckoned by most data scientists would be "explore-exploit". Pls let me know if that's what came to your mind too! This article is an attempt to introduce a new perspective to associate MAB algorithms with a different

A Simple A/B Stats Calculator for You

It’s been a while since my last post on Navigating A/B-Seas , but I was prodding myself to be back, doing the thing I enjoy the most - writing. Over the weekend, I decided to roll up my sleeves and bring to life a small side project I’ve

Experimentation & A/B Testing

You Don't Need a Crystal Ball or Calculus to Do Sizing

In my experience with data professionals, sizing is often one of the most misunderstood or misapplied efforts. This is a bold accusation, but I come from a place of atonement rather than audacity. I have done sizing the wrong way for years until I learnt the simple recipe behind the

Experimentation & A/B Testing

When You Have Counts, Expected Value Is All You Need

Is 10 and 11 significantly different from each other? What about 10,000 & 11,000. Just comparing counts is unusual in the world of experimentation, since we are often focused on comparing differences in means or differences in proportions. In many cases, this nudges us to create metrics that

Experimentation & A/B Testing

CUPED: What You Know Before the Experiment Matters as Much as What You Know After

Note : This is the third part of a series of articles discussing variance reduction techniques in A/B testing. Written in collaboration with Athulya Ganapathi Kandy The previous parts can be found here: * PART I : All strata are useful, but some strata are more useful than the others | LinkedIn * PART

Experimentation & A/B Testing

Capping Is the Low-Hanging Fruit You Should Pick, Unless the Whales Matter

Note: This is the second part of a series of articles discussing variance reduction techniques in A/B testing. The first part can be found here: All strata are useful, but some strata are more useful than the others | LinkedIn When considering variance reduction techniques, there is a very simple

Experimentation & A/B Testing

All Strata Are Useful, But Some Strata Are More Useful Than Others

Written in collaboration with Athulya Ganapathi Kandy NOTE : This is the first part of a series of posts discussing variance reduction techniques in A/B testing Introduction The biggest adversary an analytics professional encounters in their experimentation journey is (statistical) insignificance, which is the antithesis of what A/B tests

Experimentation & A/B Testing

Multiple Test Corrections: Where False Positives Are Rigorously Trimmed

When it comes to multiple test corrections in A/B tests, two methods stand out - Bonferroni Correction and Benjamini-Hochberg (B-H) Corrections. In this blog post, we will explore: * Commonly overlooked multiple testing scenarios * The difference between Bonferroni Correction and B-H Correction * Python Implementation for both methods Commonly Overlooked Multiple