- Home
- Skills
- Research & Analysis
- Experiment Design: A/B Tests That Answer
More screenshots
Works with the AI Agents you already use
Experiment Design: A/B Tests That Answer
Design A/B tests that end in a decision: a falsifiable hypothesis, a pre-committed contract, honest sample sizing, and a ship, kill, or inconclusive verdict you follow.
$5
Experiment Design: A/B Tests That Answer
Example session with this skill installed
We want to test simplifying our pricing page — one recommended plan instead of the three-tier grid. Conversion is about 12% and we get roughly 4,000 visitors a day. Design the test.
- Read your context and instructions
- Compiled the experiment design
PRE-COMMITMENT CONTRACT — produced by applying the skill.
Test: pricing page simplification. Written and timestamped before launch; stored where it cannot be edited silently.
Hypothesis: replacing the three-tier comparison grid with one recommended tier (alternatives behind a "compare plans" toggle) will lift signup-to-paid conversion 8% relative — 12.0% to about 13.0% — by cutting decision friction for visitors who already intend to subscribe. Falsifier recorded: we will not ship if the refund guardrail trips, even on a clear conversion win.
Primary metric: signup-to-paid conversion. Guardrails, each with a trip threshold: refund rate within 14 days (+0.3 points trips), support tickets per signup (+10% relative trips), page latency p75 (+200 ms trips), revenue per signup (−3% relative trips).
Sizing: MDE 1.0 percentage point at 80% power, alpha 0.05 two-sided, needs about 17,500 visitors per arm. At roughly 2,000 per arm per day that is about 9 days of sample — but UI changes carry a 14-day floor for novelty and primacy effects, so duration is set to 14 days, covering two full weekly cycles. Feasibility check passed: required sample arrives well inside the six-week ceiling.
Decision rule, fixed now: ship if the confidence interval excludes zero upward and every guardrail holds; kill if it excludes zero downward or any guardrail trips; otherwise the result is inconclusive and the default is kill — a revisit requires a bolder redesign under a new contract. No segment analysis beyond what is declared here. Analysis happens once, on day 15. Daily peeking is not analysis, and "directional" is not a verdict.
Connects securely to your tools. The creator never sees your data.
About this skill
Most experiment programs produce arguments, not answers: vague hypotheses nobody can falsify, daily peeking until something turns green, guardrails checked after the ship decision, and "directional but not significant" treated as evidence. This skill replaces all of that with one artifact — the pre-commitment contract, written before launch, timestamped somewhere it cannot be quietly edited, and followed mechanically at readout. It starts with hypothesis craft: cause, effect, magnitude, and mechanism, plus a falsifiability test that kills experiments whose outcome cannot change the decision. Then sizing as arithmetic, not hope: the inverse-square relationship between detectable effect and sample, an 80% power floor, and a feasibility decision point that stops tests whose required sample cannot arrive inside the duration ceiling — with the honest alternatives named (bolder change, wider eligibility, or a different method entirely). Duration rules cover weekly cycles, the 14-day floor for UI changes, novelty and primacy effects, and long-term holdouts for permanent changes. Running discipline covers peeking and false-positive inflation, binding guardrails with numeric trip thresholds, concurrent-test interference, ratio-metric variance, and sample-ratio mismatch as a stop signal. Readout resolves into three buckets — ship, kill, inconclusive — with ranked responses for the inconclusive case and a hard rule that post-hoc segments are noise until a confirmatory test says otherwise. You get the fill-in contract template, a fully worked pricing-page example with real sizing numbers, a design readiness rubric, a pre-launch checklist, and edge-case handling for network effects, low traffic, and contaminated runs.
- It cannot manufacture traffic: if your volume cannot support the MDE inside the duration ceiling, it will tell you the test is not worth running rather than design a weaker one quietly.
- It assumes standard frequentist analysis; teams committed to Bayesian or sequential frameworks keep the contract discipline but should swap in their own readout math.
- Network-effect products (marketplaces, social graphs) break plain user-level randomization — the skill flags this and points to cluster or geographic designs, but those need platform support it cannot provide.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 24 days ago
Permissions
No special permissions declared or detected