Experiment Designer

The Experiment Designer skill helps product teams design, prioritize, and evaluate product experiments. It guides users through hypothesis creation, metric d.

Install

npx promptshop add experiment-designer

Details

What This Skill Does

The Experiment Designer skill helps product teams design, prioritize, and evaluate product experiments.
It guides users through hypothesis creation, metric definition, sample size estimation, ICE scoring, and results interpretation to make data-driven product decisions.
This skill is ideal for product managers, data scientists, and engineers involved in A/B testing and experimentation.

When to Use

Planning A/B and multivariate experiments.
Writing clear hypotheses with measurable outcomes.
Estimating sample size for statistical significance.
Prioritizing experiments using ICE scoring.
Interpreting statistical output for product decisions.
Defining primary, guardrail, and secondary metrics.

Key Features

Generates hypotheses in If/Then/Because format.
Calculates sample size based on baseline and MDE.
Prioritizes experiments using the ICE scoring model.
Provides a hypothesis quality checklist.
Identifies common experiment pitfalls.
Defines key metrics for experiment success.

Manual Installation

Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

When To Use

Use this skill for: A/B and multivariate experiment planning Hypothesis writing and success criteria definition Sample size and minimum detectable effect planning Experiment prioritization with ICE scoring Reading statistical output for product decisions

Core Workflow

Write hypothesis in If/Then/Because format If we change [intervention] Then [metric] will change by [expected direction/magnitude] Because [behavioral mechanism]

Define metrics before running test Primary metric: single decision metric Guardrail metrics: quality/risk protection Secondary metrics: diagnostics only

Estimate sample size Baseline conversion or baseline mean Minimum detectable effect (MDE) Significance level (alpha) and power

Use: python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute

Prioritize experiments with ICE Impact: potential upside Confidence: evidence quality Ease: cost/speed/complexity

ICE Score = (Impact Confidence Ease) / 10

Launch with stopping rules Decide fixed sample size or fixed duration in advance Avoid repeated peeking without proper method Monitor guardrails continuously

Interpret results Statistical significance is not business significance Compare point estimate + confidence interval to decision threshold Investigate novelty effects and segment heterogeneity

Hypothesis Quality Checklist

[ ] Contains explicit intervention and audience [ ] Specifies measurable metric change [ ] States plausible causal reason [ ] Includes expected minimum effect [ ] Defines failure condition

Common Experiment Pitfalls

Underpowered tests leading to false negatives Running too many simultaneous changes without isolation Changing targeting or implementation mid-test Stopping early on random spikes Ignoring sample ratio mismatch and instrumentation drift Declaring success from p-value without effect-size context

Statistical Interpretation Guardrails

p-value < alpha indicates evidence against null, not guaranteed truth. Confidence interval crossing zero/no-effect means uncertain directional claim. Wide intervals imply low precision even when significant. Use practical significance thresholds tied to business impact.

See: references/experiment-playbook.md references/statistics-reference.md

Tooling

scripts/sample_size_calculator.py

Computes required sample size (per variant and total) from: baseline rate MDE (absolute or relative) significance level (alpha) statistical power

Example: python3 scripts/sample_size_calculator.py
--baseline-rate 0.10
--mde 0.015
--mde-type absolute
--alpha 0.05
--power 0.8Experiment Designer