Agent Eval Report Inspector

    1

    Inspect JSON/JSONL agent evaluation reports for pass rates, regressions, and latency metrics.

    Free

    1 useSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Eval Report Inspector

    Example session with this skill installed

    Inspect the eval report at logs/v2-alpha-run.json and give me a summary of pass rates and any regressions compared to the previous run.

    • Read your context and instructions
    • Compiled the agent eval report

    I have analyzed the evaluation report at logs/v2-alpha-run.json. The inspection identifies a slight dip in pass rates and three specific regression cases in the tool-use category.

    Agent Evaluation Summary: v2-alpha-run

    MetricValueStatus
    Total Cases450-
    Pass Rate92.4%⚠️ Down 1.2%
    Clean Passes398-
    Passes w/ Warnings18-
    Flaky Cases5-
    Regressions3❌ Action Required
    Timeouts2-
    Avg Latency1.84s✅ Optimized

    Regression Details

    • case_088_weather_api: Failed (Timeout) - Previously Passed
    • case_142_sql_injection_filter: Failed (Safety Violation) - Previously Passed
    • case_301_multi_turn_memory: Failed (Logic Error) - Previously Passed

    Latency & Safety

    • Max Latency: 12.4s in case_205_long_context_retrieval.
    • Safety Outcomes: 48/50 privacy cases passed. 2 failures triggered redaction flags.

    Next steps

    1. Review the report.jsonl logs for case_142 to inspect the safety filter logs.
    2. Investigate the timeout in case_088 to see if the mock API was unresponsive.
    3. Run the inspector with --json to export these metrics to the release dashboard.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Analyzing raw agent evaluation data is tedious and prone to human oversight. This skill automates the inspection of JSON and JSONL reports produced by evaluation harnesses to identify pass rates, regressions, and latency issues instantly.

    What it does

    • Calculate metrics providing pass rates, flaky counts, and total case totals from harness data.
    • Identify regressions by flagging cases that moved from pass to fail between runs.
    • Track performance using latency statistics to find slow-performing agent steps.
    • Surface failures by categorizing timeouts, system errors, and blocking failures.
    • Safety reporting by isolating safety and privacy case outcomes for quick review.

    How it works

    1. Load data by reading the standard report.json and its associated report.jsonl logs.
    2. Execute command using scripts/inspect_report.py followed by the target file path.
    3. Toggle output with the --json flag if you need machine-readable summaries for CI/CD pipelines.

    Frameworks & tools

    This skill works with agent evaluation harnesses that output standard JSON or JSONL formats. It uses a Python-based inspection script to parse attempt logs and metric blocks.

    Why this beats prompting it yourself

    Parsing nested JSON logs via standard prompting often leads to hallucinated percentages or missed edge cases in large files. This skill uses specific logic to ensure counts are precise and regressions are caught across thousands of lines of logs.

    Use cases

    • Pre-release audits to verify that a new agent version has no regressions.
    • Flakiness detection to find tests that pass and fail inconsistently across attempts.
    • Performance monitoring to pinpoint exactly which steps are causing high latency.
    • Safety verification to isolate and review all safety-tagged test cases.

    Known limitations

    Does not execute agents or re-run tests. It only summarizes existing evidence. It cannot infer the root cause of a failure beyond what is recorded in the logs.

    Requires existing JSON/JSONL reports. Does not execute code or follow URLs within reports. Does not provide root cause analysis for failures.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 27 days ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Listed27 days ago

    What's inside

    Frequently Asked Questions

    The inspector is compatible with any agent evaluation harness that outputs data in standard JSON or JSONL formats, making it framework-agnostic for tools that generate structured logs.

    Unlike basic prompts that can hallucinate numbers, this skill uses a specialized Python script to parse nested logs, ensuring 100% accurate pass rates and regression tracking even in files with thousands of entries.

    You will receive the core inspection script, the necessary environment configuration files, and documentation on how to integrate the tool into your existing benchmarking workflows.

    While it cannot re-run the tests, it excels at identifying regressions by comparing current results against previous runs to flag cases that moved from 'pass' to 'fail.'

    The skill provides a --json flag, allowing you to output machine-readable summaries that can be consumed by CI/CD pipelines to automatically block deployments if pass rates fall below a certain threshold.

    The inspector requires that your evaluation logs include time-stamped steps; it then calculates statistics to pinpoint specific agent actions that are causing performance bottlenecks.

    New in Testing & Debugging