Agent Eval Suite Linter

    1

    Validate AI agent evaluation suites for syntax errors, unique IDs, and mandatory safety coverage before execution.

    Free

    1 useSecurity scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Eval Suite Linter

    Example session with this skill installed

    Lint my evaluation suite located at configs/agent-bench-v1.json and provide a machine-readable report for my CI pipeline.

    • Read your context and instructions
    • Compiled the agent eval suite

    I have completed the linting process for configs/agent-bench-v1.json. The suite contains 12 test cases. I found 1 error that must be fixed before execution and 2 warnings regarding safety coverage.

    Lint Report: agent-bench-v1.json

    SeverityRule IDCase IDMessage
    ERRORERR_DUPLICATE_IDcase-004Non-unique case ID detected. Duplicate found at index 7.
    WARNINGWARN_MISSING_INJECTIONN/ASuite lacks a blocking prompt-injection test case.
    WARNINGWARN_MISSING_SECRET_ASSERTcase-009Command handles sensitive data but lacks a negative secret-output assertion.

    Next steps

    • Resolve the ERR_DUPLICATE_ID in configs/agent-bench-v1.json by assigning a unique ID to the second instance of case-004.
    • Add a test case with the risk: prompt-injection and blocking: true tags to satisfy safety coverage requirements.
    • Run python scripts/lint_suite.py configs/agent-bench-v1.json --report lint_results.json to generate the final artifact for your CI pipeline.
    agent-bench-v1.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Poorly structured evaluation suites lead to wasted compute and false positives in agent testing. This skill validates Agent Eval & Regression Harness JSON suites for structural integrity and safety coverage before you trigger expensive execution runs.

    What it does

    • Syntax validation checks for valid JSON structure and required suite metadata fields.
    • Case integrity ensures all case IDs are unique and command argument arrays are populated.
    • Safety coverage auditing identifies missing blocking prompt-injection cases or secret-output assertions.
    • Configuration checking flags invalid timeout values, non-positive repetitions, and unrecognized risk tags.
    • Duplicate detection finds and reports redundant case definitions within the suite.

    How it works

    1. Analyze suite by pointing the linter to your target suite.json file.
    2. Execute lint command using the python scripts/lint_suite.py script.
    3. Review findings categorized into release-blocking errors and review-only warnings.
    4. Export results to a machine-readable format using the --report flag for CI/CD integration.

    Frameworks & tools

    This tool is built for Python environments and validates JSON-based evaluation suites used in agentic regression testing.

    Why this beats prompting it yourself

    Manual reviews of large JSON suites often miss duplicate IDs or subtle gaps in safety coverage. This skill applies a fixed set of technical rules and coverage heuristics consistently, catching syntax errors that would cause an agent runner to crash mid-execution.

    Use cases

    • Pre-flight checks to validate suite structure before starting a long-running agent benchmark.
    • CI/CD integration to block releases if a suite lacks critical safety assertions or contains duplicate cases.
    • Safety auditing to ensure every suite includes mandatory prompt-injection and data-leakage test vectors.

    Known limitations

    This skill validates suite design only. It does not execute agents, check factuality, provide security sign-off, or verify URL accessibility within the suite.

    Validates design only. Does not execute agents, check factuality, or provide safety sign-off. Requires valid JSON input.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 27 days ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed27 days ago

    What's inside

    Frequently Asked Questions

    This skill prevents wasted compute costs and runtime crashes by identifying structural errors, duplicate case IDs, and missing safety assertions in your JSON test suites before you trigger an expensive agent execution.

    The linter is compatible with any agentic evaluation framework that utilizes JSON-based suite definitions, specifically those structured for regression harnesses and benchmarking.

    Yes, the skill includes a `--report` flag that exports findings into a machine-readable format, making it easy to automate pre-flight checks and block builds in GitHub Actions or Jenkins if a suite is invalid.

    It audits your suite for the presence of mandatory safety test vectors, such as prompt-injection attempts and secret-leakage assertions, ensuring your agent isn't deployed without rigorous vulnerability testing.

    You will receive the core linting script, a library of validation rules, documentation for CLI commands, and a set of sample valid/invalid suites to help you get started.

    This tool is a static analyzer that validates your test design; it does not execute the agent itself, verify the truthfulness of agent outputs, or check if external URLs in your suite are live.

    New in Testing & Debugging