Experimentation & A/B Testing Skill

When to Use

• Validating new subject lines or creative.
• Testing segmentation hypotheses (persona vs behavior).
• Optimizing cadence, timing, or automation triggers.

Framework

1. Hypothesis – define expected uplift + rationale.
2. Metric Selection – primary (open/click/conv) + guardrails (unsubs, spam).
3. Sample Sizing – ensure stat significance (min 500 recipients per variant or use power calculator).
4. Execution – randomize, keep variants isolated, limit simultaneous tests.
5. Analysis – use z-test or Bayesian uplift; document learnings.

Templates

• Experiment brief (hypothesis, segments, KPI, risk guardrails).
• Variant table (control vs test inputs, creative asset links, owner).
• Calculator sheet for minimum detectable effect + sample size.
• Post-test debrief doc capturing learnings + rollout plan.

Experiment Ideas

• Subject line vs preview text combos.
• CTA placement (hero vs footer).
• Personalization depth (basic vs dynamic modules).
• Wait times between touches.

Tips

• Run no more than two tests per journey simultaneously.
• Recycle learnings into playbooks + automation templates.
• Segment results by persona to catch hidden signals.