More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Experimentation Analytics: Read Results Without Fooling Yourself

    1

    Read a finished A/B test panel without fooling yourself: confidence interval first, peeking and multiple-testing discounts applied, dashboard reconciled, verdict defended.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Experimentation Analytics: Read Results Without Fooling Yourself

    Experimentation Analytics: Read Results Without Fooling Yourself

    Example session with this skill installed

    Our checkout redesign test finished: 3 weeks, 61,400 users. Conversion is +2.1% relative, p = 0.03, 95% CI [+0.2%, +4.0%]. The team checked the dashboard most days. Mobile users show +5.9% (p = 0.01). Should we ship?

    • Read your context and instructions
    • Compiled the experimentation analytics

    VERDICT: Not yet — promising, not proven. Extend to a pre-committed end date or rerun the read with sequential statistics before shipping.

    THE CI READ — [+0.2%, +4.0%] excludes zero, so the direction is probably real. But the lower bound is what you are buying: +0.2% relative on checkout conversion. Price that against the redesign's build and maintenance cost before celebrating the +2.1% headline.

    THE PEEKING PROBLEM — p = 0.03 assumes one look at the end. The team looked most days for three weeks. On a fixed-horizon test, repeated peeking inflates the false-positive rate well past 5% — a p of 0.03 after roughly fifteen looks is about what noise produces on its own. If your platform reports always-valid (sequential) p-values this objection disappears; a bare p-value almost always means fixed-horizon.

    THE SEGMENT — Mobile +5.9% (p = 0.01) was not a pre-registered cut; it is the best-looking slice found afterward. With segments, metrics, and variants multiplied together, one p ≈ 0.01 is expected by chance. Treat mobile as a hypothesis for the follow-up test, not a ship-to-mobile decision.

    WHAT WOULD CHANGE THE CALL — Two more weeks to a declared analysis date with no interim ship decisions, primary metric unchanged. If the CI still excludes zero with a lower bound worth the cost, ship on the average effect — and pre-register the mobile hypothesis in the rerun if you want to act on it.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The result panel is where good experiments go to be misread. A p-value of 0.03 gets treated as proof, a peeked-at test gets shipped, a lucky segment gets quoted in the launch thread, and the dashboard number — measuring something else entirely — gets used to overrule the experiment. This skill is the disciplined read. It starts with the confidence interval, classified into five cases that set the provisional verdict, because the interval says what you actually know and the lower bound is what you are really buying. It then prices in the validity threats most teams skip: whether the analysis was fixed-horizon or sequential and how often someone peeked, how many comparisons were actually made across variants, metrics, segments, and windows, whether variance reduction like CUPED is being read correctly, and whether ratio metrics are estimated with ratio-aware math instead of a naive t-test. Segments are read under strict rules — pre-registered cuts are evidence, post-hoc cuts are hypotheses. A reconciliation protocol handles the moment BI and the experiment disagree, using denominators, windows, and definitions instead of volume. Everything lands in a decision table, a panel intake template, and a pre-delivery quality gate, and it protects the hardest call in analytics: inconclusive is an answer, and calling it is a skill, not a failure.

    • It interprets results; it does not design the test, size the sample, or compute statistics from raw event data — bring the panel your platform produced.
    • If your platform cannot produce a confidence interval or disclose its variance method, the skill flags the gap but cannot close it.
    • Sequential and Bayesian platforms phrase outputs differently; the reading discipline transfers, but the skill's examples use the common frequentist panel.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago

    What's inside

    Frequently Asked Questions

    Yes, but the first step is to get the interval. The skill explains why a p-value alone cannot support a ship decision — it says nothing about magnitude — and what to ask your platform for instead.

    No, and the skill shows you the better question: read the interval. A p of 0.06 with a wide interval usually means inconclusive, not nearly-proven, and the honest move is more data or a bigger change, not a friendlier threshold.

    Usually both, about different things. The skill reconciles by denominator, window, external effects, and metric definition, and shows how to report the experiment number precisely for the enrolled population.

    New in Research & Analysis