More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Experiment Audit: Integrity and Honesty Checks

    1

    An adversarial integrity audit for finished experiments: ground-truth provenance, score normalization, phantom results, and scope honesty, ending in a PASS, WARN, or FAIL verdict.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Experiment Audit: Integrity and Honesty Checks

    Experiment Audit: Integrity and Honesty Checks

    Example session with this skill installed

    Audit my completed experiment before I write it up. An agent ran a retrieval study for me and reports 0.97 accuracy across a comprehensive benchmark suite. The eval scripts, result files, and tracker are in the project folder; the draft summary citing the numbers is there too.

    • Read your context and instructions
    • Compiled the experiment audit

    INTEGRITY AUDIT — retrieval study, artifacts collected mechanically (6 eval scripts, 14 result files, 1 tracker, draft summary). Reviewer independent of the run. Verdict: FAIL.

    Check A, ground-truth provenance — FAIL. The main eval script builds its "reference" answers by querying the same model family under evaluation and caching the outputs as ground truth. The 0.97 headline is agreement with the model's own outputs, reported as accuracy, with no proxy labeling in the summary or the scripts' output headers. This is the cardinal failure: the number measures self-consistency, not performance.

    Check B, score normalization — WARN. Per-dataset scores in the cited table are normalized by the best score within each run batch. Raw scores exist in the result files, but the summary cites only the normalized values, which flatters the two weakest datasets.

    Check C, result existence — FAIL. The summary's strongest comparison cites results/final_v2.json. That file does not exist; the tracker lists the run as TODO. Two numbers in the table match no key in any result file supplied.

    Check D, dead code — WARN. Two metric functions (an exact-match scorer and a latency metric) are defined in the eval package and never called; the reported metric set appears selected after the runs.

    Check E, scope honesty — WARN. "Comprehensive benchmark suite" is three datasets, one seed each. Check F, typing: synthetic proxy, mislabeled in the summary as benchmark accuracy.

    Claim impact: the 0.97 accuracy claim is unsupported; dataset-level comparisons that trace to real files need qualifiers (proxy evaluation, single seed). Fix path: re-run against the dataset's actual reference answers, cite raw scores, then re-audit the new artifacts. Rewriting the summary around the existing numbers is explicitly not a fix.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Results produced under optimization pressure fail in characteristic ways, whether the optimizer is a person chasing a deadline or an agent chasing a metric: ground truth quietly derived from the model's own outputs, scores normalized by the model's own maximum, headline numbers cited from result files that were never produced, and a two-configuration pilot described as a comprehensive evaluation. This skill is the integrity check that runs once, after experiments complete and before any claims are written. Its structure is the safeguard: the party that ran the experiment never judges it. The executor collects artifacts mechanically — evaluation scripts, result files, ground-truth sources, the experiment tracker, configs, and the claims narrative — as paths only, and an independent reviewer works under an adversarial brief: assume the evaluation is compromised somewhere, find where. Six checks cover ground-truth provenance (the cardinal failure: agreement-with-yourself reported as performance), score normalization, result existence (phantom results fail outright), dead metric code, scope honesty, and honest evaluation typing — real ground truth, synthetic proxy, simulation-only, human evaluation. The verdict is PASS, WARN, or FAIL with fixed conditions, every check anchored to file-and-line evidence, and every claim in the narrative mapped to supported, needs qualifier, or unsupported, with tags that travel downstream into review and writing stages. A report template, machine-readable verdict format, and a worked example of a FAIL verdict are included.

    • It catches common, known integrity patterns, not every possible form of misconduct; a PASS means "no known integrity failure found," never "the science is correct."
    • The audit is only as strong as the artifacts supplied — work whose scripts and result files cannot be produced is graded FAIL by absence rather than taken on trust.
    • It judges evaluation integrity, not method quality or experimental design; a sound audit verdict and a weak method can coexist.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    Because the failure mode is structural: the optimizer that produced the numbers will certify them. The protocol separates collection from judgment — the executor gathers artifact paths mechanically, and an independent reviewer, working adversarially from the files alone, issues the verdict.

    No. Proxies and simulations are legitimate when labeled. The audit fails unlabeled proxies presented as ground truth; a clearly labeled proxy evaluation gets an honest typing classification and its claims are judged against what that evidence type can support.

    No — that is the one fix the protocol forbids. A FAIL means the evaluation itself is compromised (ground truth from model outputs, phantom results, fabricated normalization). The fix is to repair or re-run the evaluation and audit the new artifacts; rewriting the narrative around bad numbers is exactly the failure the audit exists to prevent.

    New in Research & Analysis