More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Stateful Agent Evaluation & Failure Diagnosis

    1

    Design agent evaluations, verify state changes and tool calls, and diagnose failures with traceable evidence.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Stateful Agent Evaluation & Failure Diagnosis

    Stateful Agent Evaluation & Failure Diagnosis

    Example session with this skill installed

    Use Stateful Agent Eval to check the attached contract.json and evidence.json for a role-change agent. The rule is that requesting an administrator role must not grant it before approval. Verify both cases, distinguish failures from missing evidence, and deliver a report with evidence references, possible causes and the smallest useful retest. These inputs are synthetic demonstration data.

    • Read your context and instructions
    • Compiled the stateful agent evaluation

    Result: FAIL

    This is a synthetic example checked with the deterministic verifier. Scope: supplied artifacts only; no live business system was executed.

    CaseExecutionRequired toolRole unchanged on requestEnd matches startOverall
    request_then_cancelPASSPASSFAILPASSFAIL
    evidence_gapPASSPASSINCONCLUSIVEPASSINCONCLUSIVE

    Confirmed failure

    For member U-1, the submitted snapshots show viewer → admin → viewer. The requested checkpoint already grants admin, violating the contract that a request must not grant a role before approval. Returning to viewer at the end does not erase that intermediate violation.

    Evidence

    • Before: evidence.json#/attempts/0/checkpoints/0/state/memberships
    • Requested: evidence.json#/attempts/0/checkpoints/1/state/memberships
    • End: evidence.json#/attempts/0/checkpoints/2/state/memberships

    Evidence gap

    The second case is missing its required requested checkpoint (evidence.json#/attempts/1/checkpoints). Matching start and end snapshots cannot establish what happened in between. The result remains INCONCLUSIVE; it is not counted as a pass.

    Diagnosis and next step

    A request handler might be changing membership immediately, or the exporter might have mislabeled the snapshot. These are hypotheses, not established causes. Inspect the request-to-write path and checkpoint association. Capture committed state immediately after the request and before approval or cancellation, linked to tool and transaction records. Rerun both cases with the same contract and one planned attempt each.

    Delivered: report.json with check details and evidence references, report.md, and this diagnosis. Summary: 0 PASS, 1 FAIL, 1 INCONCLUSIVE, 0 ERROR. Baseline eligibility is not established. Real approver identity, production behavior, concurrency and natural-language answer quality remain outside this result.

    report.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Evaluate what your agent actually changed, with evidence you can inspect.

    Built for developers working on agents with tool calls, approvals and business side effects: cancellations, refunds, coupon issuance, purchasing requests and similar workflows.

    What you can do

    • Design an evaluation: turn established business rules into scenarios, a structured contract and a practical evidence-collection plan. Missing rules stay explicit rather than becoming invented expectations.
    • Verify a run: check supplied tool traces and committed state snapshots against that contract using a standalone deterministic verifier.
    • Diagnose failures: separate observed violations from possible causes, identify missing evidence and choose a focused retest.

    What the checks cover

    Exact record counts, expected state changes, protected objects, tool arguments and errors, record relationships, audit-event order and execution status. Detect duplicate effects and recorded changes before approval, even when the final state looks correct.

    Reports distinguish PASS, FAIL, INCONCLUSIVE and ERROR, with expected and observed values and evidence references. Missing cases and planned attempts remain visible instead of disappearing from the result.

    Inputs and deliverables

    For design, provide your business rules, tool definitions, state schema and approval boundaries. Receive an evaluation plan, a draft contract and unresolved questions.

    For verification, provide contract.json and evidence.json containing traces, snapshots and declared collection coverage. Receive report.json, readable check results and an evidence-led diagnosis with concrete next steps. A dedicated adapter also accepts complete Commerce Agent 2.0 reports; other projects use the documented generic format.

    Included in v0.1.0

    The Skill workflow, input-format and diagnosis references, a Python verifier, the Commerce report adapter, six synthetic commerce scenarios and a procurement example. The implementation passes 76 local automated tests, including negative cases and missing-evidence checks. These tests are not agent-quality scores.

    Requirements and scope

    Deterministic verification requires Python 3.10+ and file/terminal access. Python standard library only; no additional packages, database connection or model API key is needed by the scripts. Your own test harness runs the target agent and exports the evidence.

    The Skill checks supplied artifacts; it does not execute your agent or business services. It does not certify real approver identity, concurrency or natural-language answer quality. If script execution is unavailable, it provides design or clearly labeled manual analysis, without claiming a verified PASS.

    Source is provided under the included MIT license. The package has been tested locally; Agensi runtime verification is pending.

    Checks only supplied structured evidence; the target agent and business services must be run by your own harness. Missing checkpoints or incomplete collection coverage remain INCONCLUSIVE. The Commerce 2.0 adapter does not yet verify required/forbidden knowledge-source sets. Recorded snapshots cannot prove the absence of changes between observations, real human authorization, concurrency or database locking. Natural-language answer quality is outside the deterministic checks. Python 3.10+ is required for verification; without it, only design or labeled manual analysis is available. Local tests passed; Agensi runtime verification is pending.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 23 days ago

    Requires

    Python 3.10+
    Terminal access
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    **/*.json
    **/*.md
    **/scripts/*.py

    Reads only user-selected rules, contracts, evidence and relevant source files. Runs the bundled Python scripts from temporary storage and writes evaluation plans and reports to the user's chosen output location. Staged Skill source is removed after the task. No credentials, connected accounts, live business services or network access are required by the verification scripts.

    Listed23 days ago

    What's inside

    Frequently Asked Questions

    For evaluation design, provide established business rules, tool definitions, your state schema and approval boundaries. For deterministic verification, provide contract.json and evidence.json with tool events, committed snapshots and collection coverage in the documented format. The package includes examples and an adapter for complete Commerce Agent 2.0 reports.

    No. Your own test harness executes the target agent and collects committed state and tool evidence. This Skill designs the checks, verifies the supplied artifacts offline and helps explain failures. The Python scripts do not connect to live business services, issue refunds or require API credentials.

    Yes. The core format lets you declare your own collections, stable keys, tool names, money fields and rules. A procurement example demonstrates a different domain. v0.1 includes one dedicated importer for Commerce Agent 2.0 reports; other frameworks need an explicit mapping into the generic contract and evidence format.

    Missing checkpoints, incomplete collection coverage or unsupported obligations remain INCONCLUSIVE and prevent a full PASS. A confirmed violation is still reported as FAIL alongside any gaps. If Python 3.10+ cannot run, the Skill can help design a contract or provide clearly labeled manual analysis, but it will not claim that deterministic verification was executed.

    New in Testing & Debugging