- Home
- Skills
- Testing & Debugging
- Stateful Agent Evaluation & Failure Diagnosis
More screenshots
Works with the AI Agents you already use
Stateful Agent Evaluation & Failure Diagnosis
Design agent evaluations, verify state changes and tool calls, and diagnose failures with traceable evidence.
$5
Stateful Agent Evaluation & Failure Diagnosis
Example session with this skill installed
Use Stateful Agent Eval to check the attached contract.json and evidence.json for a role-change agent. The rule is that requesting an administrator role must not grant it before approval. Verify both cases, distinguish failures from missing evidence, and deliver a report with evidence references, possible causes and the smallest useful retest. These inputs are synthetic demonstration data.
- Read your context and instructions
- Compiled the stateful agent evaluation
Result: FAIL
This is a synthetic example checked with the deterministic verifier. Scope: supplied artifacts only; no live business system was executed.
| Case | Execution | Required tool | Role unchanged on request | End matches start | Overall |
|---|---|---|---|---|---|
| request_then_cancel | PASS | PASS | FAIL | PASS | FAIL |
| evidence_gap | PASS | PASS | INCONCLUSIVE | PASS | INCONCLUSIVE |
Confirmed failure
For member U-1, the submitted snapshots show viewer → admin → viewer. The requested checkpoint already grants admin, violating the contract that a request must not grant a role before approval. Returning to viewer at the end does not erase that intermediate violation.
Evidence
- Before:
evidence.json#/attempts/0/checkpoints/0/state/memberships - Requested:
evidence.json#/attempts/0/checkpoints/1/state/memberships - End:
evidence.json#/attempts/0/checkpoints/2/state/memberships
Evidence gap
The second case is missing its required requested checkpoint (evidence.json#/attempts/1/checkpoints). Matching start and end snapshots cannot establish what happened in between. The result remains INCONCLUSIVE; it is not counted as a pass.
Diagnosis and next step
A request handler might be changing membership immediately, or the exporter might have mislabeled the snapshot. These are hypotheses, not established causes. Inspect the request-to-write path and checkpoint association. Capture committed state immediately after the request and before approval or cancellation, linked to tool and transaction records. Rerun both cases with the same contract and one planned attempt each.
Delivered: report.json with check details and evidence references, report.md, and this diagnosis. Summary: 0 PASS, 1 FAIL, 1 INCONCLUSIVE, 0 ERROR. Baseline eligibility is not established. Real approver identity, production behavior, concurrency and natural-language answer quality remain outside this result.
Connects securely to your tools. The creator never sees your data.
About this skill
Evaluate what your agent actually changed, with evidence you can inspect.
Built for developers working on agents with tool calls, approvals and business side effects: cancellations, refunds, coupon issuance, purchasing requests and similar workflows.
What you can do
- Design an evaluation: turn established business rules into scenarios, a structured contract and a practical evidence-collection plan. Missing rules stay explicit rather than becoming invented expectations.
- Verify a run: check supplied tool traces and committed state snapshots against that contract using a standalone deterministic verifier.
- Diagnose failures: separate observed violations from possible causes, identify missing evidence and choose a focused retest.
What the checks cover
Exact record counts, expected state changes, protected objects, tool arguments and errors, record relationships, audit-event order and execution status. Detect duplicate effects and recorded changes before approval, even when the final state looks correct.
Reports distinguish PASS, FAIL, INCONCLUSIVE and ERROR, with expected and observed values and evidence references. Missing cases and planned attempts remain visible instead of disappearing from the result.
Inputs and deliverables
For design, provide your business rules, tool definitions, state schema and approval boundaries. Receive an evaluation plan, a draft contract and unresolved questions.
For verification, provide contract.json and evidence.json containing traces, snapshots and declared collection coverage. Receive report.json, readable check results and an evidence-led diagnosis with concrete next steps. A dedicated adapter also accepts complete Commerce Agent 2.0 reports; other projects use the documented generic format.
Included in v0.1.0
The Skill workflow, input-format and diagnosis references, a Python verifier, the Commerce report adapter, six synthetic commerce scenarios and a procurement example. The implementation passes 76 local automated tests, including negative cases and missing-evidence checks. These tests are not agent-quality scores.
Requirements and scope
Deterministic verification requires Python 3.10+ and file/terminal access. Python standard library only; no additional packages, database connection or model API key is needed by the scripts. Your own test harness runs the target agent and exports the evidence.
The Skill checks supplied artifacts; it does not execute your agent or business services. It does not certify real approver identity, concurrency or natural-language answer quality. If script execution is unavailable, it provides design or clearly labeled manual analysis, without claiming a verified PASS.
Source is provided under the included MIT license. The package has been tested locally; Agensi runtime verification is pending.
Checks only supplied structured evidence; the target agent and business services must be run by your own harness. Missing checkpoints or incomplete collection coverage remain INCONCLUSIVE. The Commerce 2.0 adapter does not yet verify required/forbidden knowledge-source sets. Recorded snapshots cannot prove the absence of changes between observations, real human authorization, concurrency or database locking. Natural-language answer quality is outside the deterministic checks. Python 3.10+ is required for verification; without it, only design or labeled manual analysis is available. Local tests passed; Agensi runtime verification is pending.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 23 days ago
Requires
Permissions
File Scopes
Reads only user-selected rules, contracts, evidence and relevant source files. Runs the bundled Python scripts from temporary storage and writes evaluation plans and reports to the user's chosen output location. Staged Skill source is removed after the task. No credentials, connected accounts, live business services or network access are required by the verification scripts.