Pre-Declared ML Experiments and Certified Decision Gates

    1

    This kit is the paperwork and the arithmetic that close both: declare, run, read once, adopt or stop.

    $35

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Pre-Declared ML Experiments and Certified Decision Gates

    Example session with this skill installed

    Our re-ranker picks a product match when the inlier count is over 20, but that number was tuned by eye. Give me a certified threshold with a real error bound, and a way to stop my team from moving the goalposts after each run.

    • Read your context and instructions
    • Compiled the pre-declared ml experiments

    Summary

    Certified an inlier floor of 40 (not 20) at 5% target risk, 95% confidence, on a one-shot read of the test half; and put the predeclaration and result paperwork in front of every future run.

    Certification

    python scripts/experiment_gate.py certify --rows rows.jsonl --ladder 20,30,40,50,60,80,100 --target-risk 0.05 --confidence 0.95 --min-coverage 0.20 --lock outputs/v12-test-read.lock.json --output outputs/v12-summary.json

    RungReleased (calibration)ErrorsBonferroni bound
    201,1021280.134
    30868610.089
    40701220.047
    50590140.040

    Loosest passing rung
    40. Test half read once at 40: 688 released of 1,497 kept (coverage 0.46), 19 errors, 95% bound 0.041.

    Adopted. The lock file now refuses any second read with another floor, bar or set (exit 2).

    Paperwork

    • docs/V12_PREDECLARATION.md committed before the run (hash 3f9c1a2): the change, two arms on the frozen 3,004-photo set, bar "+3 points, at most half as many to-wrong as to-right", four predictions.
    • docs/V12_RESULT.md: cites 3f9c1a2, verdicts on all four predictions (two wrong), "Not adopted" for arm B, "Nothing since the declaration" in the code section.
    • python scripts/experiment_gate.py check-docs --predeclaration ... --result ... -> ok.

    Tests

    python -m unittest test_experiment_gate -v: 10 OK.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    After enough experiments every team discovers the same two ways to lie to itself: tuning a threshold on the set that then confirms it, and writing the bar after seeing the result. This kit is the paperwork and the arithmetic that close both: declare, run, read once, adopt or stop. It comes from a run of more than a hundred numbered experiments on a product-identification model, where the rules below were each learned by breaking them.

    Paperwork (templates/): a predeclaration (the change, the arms on one frozen set, the bar as numbers, predictions that can be wrong) committed before any arm runs; a result that cites the predeclaration's hash in its first paragraph, gives every prediction a verdict including the wrong ones, says adopted or not, and discloses what changed in the code since the declaration. check-docs fails a result that skips any of that.

    Arithmetic (experiment_gate.py, standard library): a binomial upper bound by bisection (tested against the closed form and the exact sum); a fixed hash split of query ids into calibration and test halves; a ladder of floors walked on the calibration half with the 1 - confidence split across the ladder (Bonferroni), the loosest rung at or under the target risk chosen; one read of the test half at that floor, enforced by a lock file that refuses a second read with a different floor, bar or set; adopted only when the test bound and coverage clear the bar; counts-only JSON summaries; exit 0 adopted, 1 not, 2 refused.

    Rules (reference/rules.md): hard boundaries (frozen holdouts, no same-image validation, versioned annotations, no data in Git), the eight selection conditions and when to reject, certified-floor method, seven common bugs with fixes, the before-and-after reproducibility checklist, what not to do next.

    10 offline tests. Python 3.9+, no packages, no network.

    Requires independent queries. Floors outside the pre-defined ladder are not certified. The checker validates structure and presence of sections, not the scientific quality of the content.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 19 days ago

    Requires

    Python 3.9+
    Git
    Your model and evaluation set

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Shell to run the Python script and tests; read files for rows.jsonl and the two documents; write files for the summary JSON and the test-read lock file. No network access, no environment variables.

    Listed19 days ago

    What's inside

    Frequently Asked Questions

    This skill enforces scientific rigor by requiring experimental designs to be committed to version control before execution, effectively preventing 'p-hacking' and the tuning of thresholds based on observed test results.

    The skill includes a standalone Python-based certification suite, a Markdown-based declaration framework, and a document validation tool for auditing final reports.

    Yes, it is built on standard-library Python 3.9+ with no external dependencies, making it compatible with any AI agent or workflow capable of running Python scripts and accessing Git.

    The tool uses a 'locked test read' protocol where the test data is read exactly once; if your data contains correlated queries, you must group them into single units before processing to maintain statistical validity.

    It uses a ladder of pre-declared thresholds combined with Bonferroni-corrected binomial error bounds, calculated via bisection and validated against exact binomial sums for maximum precision.

    New in LLM Ops