More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Reliability Pack — Find Where Your Agent Fails

    1

    Reads your run traces and reports task success, recovery, retry amplification, loops and the top failure classes, each with the fix that actually applies to it.

    $19

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Reliability Pack — Find Where Your Agent Fails

    Agent Reliability Pack — Find Where Your Agent Fails

    Example session with this skill installed

    python3 scripts/reliability_scan.py traces.jsonl --report reliability-report.md

    Trace rows
    {"run_id":"run-031","step":2,"tool":"search","status":"error","error":"timeout","retry":true,"latency_ms":30000}

    • Read your context and instructions
    • Compiled the agent reliability

    agent-reliability-pack 56 runs · 258 steps
    success 58.9% recovery 78.3% retry amp 1.13× loops 2
    findings 10 high 4 medium 5 low 1

    HIGH R1 success rate 58.9% of 56 runs ended without an error step
    HIGH R3 loop run-053: search called 3× with the same arguments
    HIGH R4 timeout 8 errors · search
    MEDIUM R11 retry amplification 26 retries for 23 errors (1.1×)
    MEDIUM R12 unbounded run run-055: 45 steps (cap 40)
    MEDIUM R5 rate limit 5 errors · model

    FAIL findings at or above 'high'
    report reliability-report.md

    What to do next

    1. Fix the timeout first: it is 35% of the errors and it has a retry path already.
    2. Add a repeated-action cap; both loop runs completed without an error, so nothing flagged them.
    3. Retry only transient classes, and keep one retry owner rather than two layers each retrying.
    4. Cap steps per run at the orchestrator, not in the prompt.
    5. Watch recovery after each change; it should rise before success does.
    report reliability-report.md

    Connects securely to your tools. The creator never sees your data.

    About this skill

    A demo proves an agent can finish once. Production asks whether it finishes after a timeout, a duplicate event, a tool outage or a model change — and your traces already contain the answer. This reads them and turns a pile of JSONL into a ranked reliability review.

    It classifies every failed step into a named class, then measures the things that actually separate a flaky agent from a broken one:

    • Task success and recovery rate, reported together so a retry policy cannot hide a failure
    • Retry amplification, which catches retrying errors that will never change
    • Loops: the same tool called three times with identical arguments, with the run and the arguments
    • Unbounded runs that pass the step ceiling, the denial-of-wallet failure
    • Failure counts by class and by tool, so the fix goes to the right owner
    • Latency p50 and p95 alongside the failures, because a timeout that "recovers" still ships slowly

    Each finding pairs the diagnosis with the fix that fits it: retry transient classes only, return tool errors as tool messages so the model can adapt, repair a schema error once and escalate, compact instead of retrying a context overflow, cap loops in the orchestrator rather than asking the model to stop.

    Runs on the standard library. Point it at traces, get a Markdown report and JSON, and gate CI with --fail-on high.

    Good to know: the trace says a run ended without an error step; it does not say the answer was right. Recovery is inferred from a later non-error step, not a verified final state, and error classification is keyword-based on your error field, so normalise your codes before relying on the counts. This is a diagnosis tool, not an eval — pair it with outcome evals before quoting a success rate. The shipped sample is deliberately failure-rich: 56 runs, 23 errors, 58.9% success, 2 loops.

    Success is inferred from the absence of error steps, not output correctness. Recovery is inferred from later non-error steps rather than verified final state. Requires JSONL input.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    Requires

    Python 3.10+
    A JSONL trace of agent runs

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed1 day ago

    What's inside

    Frequently Asked Questions

    The skill acts as an audit layer that analyzes your JSONL trace logs to identify specific failure modes like execution loops, retry amplification, and context overflows that manual inspection often misses.

    This tool is framework-agnostic; as long as your agent can export execution steps into a JSONL format with basic fields like run_id and status, it can analyze the data regardless of whether you use LangChain, AutoGPT, or a custom stack.

    The pack is designed for high efficiency with zero external dependencies, utilizing a Python 3.10+ CLI script that runs locally on your machine without requiring additional LLM API calls or keys.

    You will receive the core reliability_scan.py analysis script, documentation on the required JSONL schema, and a template for the markdown reports generated by the tool.

    Yes, the tool provides specific metrics on recovery rates and failure classes, allowing you to differentiate between transient errors that deserve a retry and persistent errors where retries just waste your budget.

    The current version focuses on structural and execution reliability (loops, timeouts, and errors); it does not evaluate the factual accuracy or 'hallucination' rate of the agent's final answer.

    New in LLM Ops