- Home
- Skills
- Testing & Debugging
- Agent Eval Suite Linter
Agent Eval Suite Linter
Validate AI agent evaluation suites for syntax errors, unique IDs, and mandatory safety coverage before execution.
Free
Works with the AI Agents you already use
Agent Eval Suite Linter
Example session with this skill installed
Lint my evaluation suite located at configs/agent-bench-v1.json and provide a machine-readable report for my CI pipeline.
- Read your context and instructions
- Compiled the agent eval suite
I have completed the linting process for configs/agent-bench-v1.json. The suite contains 12 test cases. I found 1 error that must be fixed before execution and 2 warnings regarding safety coverage.
Lint Report: agent-bench-v1.json
| Severity | Rule ID | Case ID | Message |
|---|---|---|---|
| ERROR | ERR_DUPLICATE_ID | case-004 | Non-unique case ID detected. Duplicate found at index 7. |
| WARNING | WARN_MISSING_INJECTION | N/A | Suite lacks a blocking prompt-injection test case. |
| WARNING | WARN_MISSING_SECRET_ASSERT | case-009 | Command handles sensitive data but lacks a negative secret-output assertion. |
Next steps
- Resolve the
ERR_DUPLICATE_IDinconfigs/agent-bench-v1.jsonby assigning a unique ID to the second instance ofcase-004. - Add a test case with the
risk: prompt-injectionandblocking: truetags to satisfy safety coverage requirements. - Run
python scripts/lint_suite.py configs/agent-bench-v1.json --report lint_results.jsonto generate the final artifact for your CI pipeline.
Connects securely to your tools. The creator never sees your data.
About this skill
Poorly structured evaluation suites lead to wasted compute and false positives in agent testing. This skill validates Agent Eval & Regression Harness JSON suites for structural integrity and safety coverage before you trigger expensive execution runs.
What it does
- Syntax validation checks for valid JSON structure and required suite metadata fields.
- Case integrity ensures all case IDs are unique and command argument arrays are populated.
- Safety coverage auditing identifies missing blocking prompt-injection cases or secret-output assertions.
- Configuration checking flags invalid timeout values, non-positive repetitions, and unrecognized risk tags.
- Duplicate detection finds and reports redundant case definitions within the suite.
How it works
- Analyze suite by pointing the linter to your target
suite.jsonfile. - Execute lint command using the
python scripts/lint_suite.pyscript. - Review findings categorized into release-blocking errors and review-only warnings.
- Export results to a machine-readable format using the
--reportflag for CI/CD integration.
Frameworks & tools
This tool is built for Python environments and validates JSON-based evaluation suites used in agentic regression testing.
Why this beats prompting it yourself
Manual reviews of large JSON suites often miss duplicate IDs or subtle gaps in safety coverage. This skill applies a fixed set of technical rules and coverage heuristics consistently, catching syntax errors that would cause an agent runner to crash mid-execution.
Use cases
- Pre-flight checks to validate suite structure before starting a long-running agent benchmark.
- CI/CD integration to block releases if a suite lacks critical safety assertions or contains duplicate cases.
- Safety auditing to ensure every suite includes mandatory prompt-injection and data-leakage test vectors.
Known limitations
This skill validates suite design only. It does not execute agents, check factuality, provide security sign-off, or verify URL accessibility within the suite.
Validates design only. Does not execute agents, check factuality, or provide safety sign-off. Requires valid JSON input.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 27 days ago
- Free with an Agensi account