Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Data Agent Eval Factory

    0

    Generate, execute, and score deterministic evaluations for SQL, RAG, and tool-calling agents.

    $8

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Data Agent Eval Factory

    Data Agent Eval Factory

    Example session with this skill installed

    Generate and run a 20-case eval for a SQL copilot using schema.sql. The agent translates natural language to Postgres. Sandbox command is 'node ./dist/runner.js'.

    • Read your context and instructions
    • Compiled the data agent eval

    I have generated 20 test cases, executed them against your local runner, and scored the results. The evaluation includes 6 adversarial cases (30%) testing for unauthorized schema modifications and prompt injections.

    SQL Agent Evaluation Report

    MetricValueResult
    Total Cases20—
    Pass Rate85%17/20
    Adversarial Pass Rate100%6/6
    SQL Correctness80%12/15
    Safety Refusals100%Correctly blocked DROP TABLE attempts
    RecommendationCONDITIONAL RELEASERequires review of complex JOIN failures

    Top Failure Analysis

    Case #14 (High): Failed to correctly join orders and line_items for "total revenue by category" query. SQL hallucinated a non-existent category_id on the orders table.

    • Case #18 (Medium): Syntax error on Postgres-specific ILIKE operator; agent used standard LIKE instead.

    Generated Artifacts

    • evals/cases_20231027.jsonl: Full test suite.
    • evals/results_20231027.jsonl: Raw agent responses.
    • evals/scores_20231027.json: Weighted scoring breakdown.

    Next steps

    1. Review the generated report.md for specific SQL join failure patterns.
    2. Refine the agent system prompt to prioritize Postgres-specific operators.
    3. Re-run the eval suite after fixes to confirm the 100% pass rate on core queries.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Data agents often fail silently through SQL hallucinations, incorrect tool selection, or unsafe data mutations. Manual testing is too slow to catch edge cases, and generic prompt lists don't validate execution accuracy or safety boundaries.

    What it does

    • Generates balanced JSONL test suites for SQL, RAG, and tool-calling agents.
    • Includes a 30% adversarial mix to test prompt injection and boundary refusals.
    • Executes deterministic evaluations against local sandboxes via stdin/stdout.
    • Scores observable agent behavior and classifies failures by severity.
    • Produces automated reports with release recommendations based on pass rates.

    Frameworks & tools

    Python, JSONL datasets, and local sandbox runners. Compatible with SQL databases, RAG systems, and tool-calling LLM architectures.

    Why this beats prompting it yourself

    Writing individual test cases is tedious and lacks coverage. This skill applies structured failure taxonomies to ensure your agent handles joins, retrieval grounding, and permission escalation attempts correctly every time, not just in happy-path scenarios.

    Use cases

    • Regression testing for SQL copilots after schema updates.
    • Evaluating RAG grounding to prevent hallucinations in customer support bots.
    • Validating tool-calling logic against adversarial prompt injections.
    • Generating release-readiness reports for stakeholder sign-off.

    Known limitations

    Requires a local sandbox command for execution mode. Does not support production writes or external side effects due to safety constraints.

    Requires a safe local sandbox command for the execution phase. Execution mode is strictly limited to non-mutating actions to prevent data loss.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Requires

    Python

    Permissions

    Terminal / Shell
    Network Access
    Environment Variables
    Listed2 months ago

    What's inside

    Frequently Asked Questions

    This skill automates the creation of high-coverage JSONL test suites and provides a framework to execute these tests against your agents, scoring them for accuracy, safety, and hallucinations rather than just output formatting.

    The skill is framework-agnostic and works with any Python-based SQL, RAG, or tool-calling agent that can be interfaced via a local sandbox command or standard I/O.

    Each generated test suite includes a 30% adversarial mix specifically designed to trigger edge cases like SQL injection, permission escalation, and retrieval grounding failures that manual testing often misses.

    You will receive the core evaluation engine, a library of failure taxonomies, templates for test suite generation, and automated reporting scripts to generate release-readiness documents.

    To maintain safety and prevent accidental data mutations, the skill requires a local sandbox runner; it is not designed to perform live writes to production databases or external APIs.

    New in Data & Databases