Senior ML Engineer, Experimentation & A/B Testing

Every launch decision at our company goes through experiment analysis, and the numbers behind those calls have to be right. We're hiring a senior ML engineer to own the offline analysis engine that reads a table of per-arm metrics and turns it into a clear ship-or-hold recommendation. This is applied statistics work, and getting the math correct matters more than anything flashy.

What you'll do:

• Compute lift and pick the correct significance test for each metric type, from proportion tests for conversion rates to Welch's t-test for continuous means
• Build seeded bootstrap confidence intervals and a rank-based test, and get the variance math right for differences in means
• Add the guardrails that keep a bad experiment from shipping, including a sample-ratio-mismatch check and a clear decision rule
• Make every run reproducible, with all randomness seeded and no surprises between runs

What we're looking for:

• A strong grounding in applied statistics: hypothesis testing, confidence intervals, bootstrapping, variance estimation
• Fluency in Python and numpy, with real discipline around reproducibility
• The judgment to make honest calls about one- versus two-sided tests, peeking, and multiple comparisons

Nice to have:

Built or worked on an A/B testing platform, and familiar with scipy and experimental design