- Home
- Skills
- Data & Databases
- Data Agent Eval Factory
Works with the AI Agents you already use
Data Agent Eval Factory
Generate, execute, and score deterministic evaluations for SQL, RAG, and tool-calling agents.
$8
Data Agent Eval Factory
Example session with this skill installed
Generate and run a 20-case eval for a SQL copilot using schema.sql. The agent translates natural language to Postgres. Sandbox command is 'node ./dist/runner.js'.
- Read your context and instructions
- Compiled the data agent eval
I have generated 20 test cases, executed them against your local runner, and scored the results. The evaluation includes 6 adversarial cases (30%) testing for unauthorized schema modifications and prompt injections.
SQL Agent Evaluation Report
| Metric | Value | Result |
|---|---|---|
| Total Cases | 20 | — |
| Pass Rate | 85% | 17/20 |
| Adversarial Pass Rate | 100% | 6/6 |
| SQL Correctness | 80% | 12/15 |
| Safety Refusals | 100% | Correctly blocked DROP TABLE attempts |
| Recommendation | CONDITIONAL RELEASE | Requires review of complex JOIN failures |
Top Failure Analysis
Case #14 (High): Failed to correctly join orders and line_items for "total revenue by category" query. SQL hallucinated a non-existent category_id on the orders table.
- Case #18 (Medium): Syntax error on Postgres-specific
ILIKEoperator; agent used standardLIKEinstead.
Generated Artifacts
evals/cases_20231027.jsonl: Full test suite.evals/results_20231027.jsonl: Raw agent responses.evals/scores_20231027.json: Weighted scoring breakdown.
Next steps
- Review the generated
report.mdfor specific SQL join failure patterns. - Refine the agent system prompt to prioritize Postgres-specific operators.
- Re-run the eval suite after fixes to confirm the 100% pass rate on core queries.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Data agents often fail silently through SQL hallucinations, incorrect tool selection, or unsafe data mutations. Manual testing is too slow to catch edge cases, and generic prompt lists don't validate execution accuracy or safety boundaries.
What it does
- Generates balanced JSONL test suites for SQL, RAG, and tool-calling agents.
- Includes a 30% adversarial mix to test prompt injection and boundary refusals.
- Executes deterministic evaluations against local sandboxes via stdin/stdout.
- Scores observable agent behavior and classifies failures by severity.
- Produces automated reports with release recommendations based on pass rates.
Frameworks & tools
Python, JSONL datasets, and local sandbox runners. Compatible with SQL databases, RAG systems, and tool-calling LLM architectures.
Why this beats prompting it yourself
Writing individual test cases is tedious and lacks coverage. This skill applies structured failure taxonomies to ensure your agent handles joins, retrieval grounding, and permission escalation attempts correctly every time, not just in happy-path scenarios.
Use cases
- Regression testing for SQL copilots after schema updates.
- Evaluating RAG grounding to prevent hallucinations in customer support bots.
- Validating tool-calling logic against adversarial prompt injections.
- Generating release-readiness reports for stakeholder sign-off.
Known limitations
Requires a local sandbox command for execution mode. Does not support production writes or external side effects due to safety constraints.
Requires a safe local sandbox command for the execution phase. Execution mode is strictly limited to non-mutating actions to prevent data loss.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Requires