Works with the AI Agents you already use
ab test design and evaluation
This skill automates the statistical rigor required for two-arm randomized controlled trials, ensuring your A/B tests are mathematically sound from design to evaluation.
$2.99
ab test design and evaluation
Example session with this skill installed
Design a two-arm test for a new landing page. Baseline conversion is 12%. I want to detect a 5% relative MDE with 80% power at 0.05 alpha. We get 5000 eligible units per day. Dataset ref: E1-LP-TEST.
- Read your context and instructions
- Compiled the ab test design
Analysis of the requested experiment design is complete. The sizing requirements and run duration have been calculated.
A/B Test Report — Landing Page Conversion Optimization, 2023-11-24
1. Scope, sources and assumptions
- Sources: E1 (User supplied design parameters).
- Dataset Reference: E1-LP-TEST (Approved by: user).
- Randomization Unit: user_id.
- Primary Metric: landing_page_conversion (type: proportion, direction: increase).
- Assumptions: Baseline value of 0.12 is treated as the true control value.
2. Experiment plan
| Parameter | Value |
|---|---|
| Units per Arm | 134,812 |
| Total Units Required | 269,624 |
| Alpha (per look) | 0.0500 |
| Power | 0.8000 |
| Absolute MDE | 0.0060 (5.0% relative) |
| Estimated Run Length | 54 days |
3. Randomization and validity audit
No results to report. Randomization audit requires inputs.results which were not supplied for this sizing request.
4. Treatment effect and guardrails
No results to report. Effect evaluation was not requested.
5. Peeking, predeclaration and data-access checks
- Looks: 0 observed / 1 planned.
- Predeclared Endpoints: landing_page_conversion (confirmed).
- Access Scope: E1-LP-TEST (Authorized).
6. Warnings and information gaps
- Information gap — organizational validation required: Deployment feasibility for 54-day duration.
7. Decision support and items for human review
- Analytical estimate — validation required: The required duration of 54 days may exceed typical sprint cycles.
- Management decision required: Approve experiment start based on 270k unit requirement.
- Proposed Action: Proceed with implementation if 54-day window is acceptable.
This analysis supports, and does not replace, professional judgment and management's decision.
Connects securely to your tools. The creator never sees your data.
About this skill
Stop guessing at experiment significance or manual sample size calculations. This skill automates the statistical rigor required for two-arm randomized controlled trials, ensuring your A/B tests are mathematically sound from design to evaluation. It handles the heavy lifting of power analysis and post-test auditing through a deterministic execution environment.
What it does
- Size experiments by computing required sample sizes and power for proportion or mean metrics based on your MDE.
- Audit randomization via sample ratio mismatch (SRM) checks, chi-square tests, and covariate balance verification.
- Detect validity issues such as data leakage, contamination, peeking, and missing consent exclusions.
- Evaluate effects by calculating treatment lift, confidence intervals, and p-values for primary and guardrail metrics.
- Check predeclarations to ensure analysis aligns with original experiment plans and look-schedules.
How it works
- Prepare a
request.jsoncontaining your experiment objective, metrics, baseline data, and dataset approval status. - The agent runs a deterministic Python script to perform all arithmetic and statistical tests.
- Review the result envelope which flags warnings, SRM failures, or power deficiencies.
- Receive a structured Markdown report containing design tables, validity audits, and causal interpretations.
Frameworks & tools
The skill relies on a self-contained Python execution environment using a specialized script for exact statistical computation. It interfaces with JSON schemas for input validation and generates GitHub-flavored Markdown reports.
Why this beats prompting it yourself
Generic LLM prompts often hallucinate p-values or fail at complex power calculations. This skill uses a deterministic script for arithmetic, ensuring every number is traceable, unrounded, and mathematically accurate according to experiment design standards.
Use cases
- Determine how many days a new checkout flow needs to run to detect a 2% lift in conversion.
- Audit a completed marketing test to see if a lopsided sample ratio invalidated the results.
- Compare treatment vs. control metrics while monitoring guardrail targets for regressions.
- Verify if "peeking" at data early has compromised the effective alpha of your experiment.
Known limitations
Does not support multi-arm, factorial, bandit, or Bayesian analyses. Limited to two-arm experiments using proportion or mean metrics. Requires pre-approved dataset access and explicit MDE inputs.
Supports only two-arm tests. No Bayesian, multi-arm, or factorial support. Requires deterministic Python script execution. No ratio-metric analysis.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 days ago