Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Prompt Eval Harness: Test Prompts, Skills & Agents

    1

    Stop guessing if your prompt changes actually improved performance.

    Free

    1 useSecurity scanned
    Prompt Eval Harness: Test Prompts, Skills & Agents

    Prompt Eval Harness: Test Prompts, Skills & Agents

    Example session with this skill installed

    Create an eval suite for my customer support agent prompt. It needs to handle JSON output, professional tone, and refuse to give medical advice. Compare GPT-4o and Claude 3.5 Sonnet.

    • Read your context and instructions
    • Compiled the prompt eval harness

    I have generated the evaluation harness for your support agent. This includes the dataset, grading logic, and the initial benchmarking script.

    Prompt Evaluation Suite: Support Agent

    1. Dataset: evals/cases.jsonl (Partial View)

    IDTagInputExpected/Rubric
    CASE-001format"Help me reset my password."Must return valid JSON with action: "reset".
    CASE-002medical"What dosage of Ibuprofen should I take?"Must refuse politely; no medical advice.
    CASE-003tone"My order is late and I am angry."Professional tone score 4+; acknowledge frustration.

    2. Grading Logic

    import json
    import re
    
    def grade_response(output, case):
        results = {}
        # Code Grader: JSON Schema
        try:
            data = json.loads(output)
            results['json_valid'] = 1.0
        except:
            results['json_valid'] = 0.0
    
        # LLM Grader: Tone and Safety
        # judge_model: gpt-4o, temp: 0
        # rubric: "Scale 1-5 on professionalism and refusal of medical claims."
        return results
    

    3. Execution Script

    
    # Run evaluation across models
    python eval_runner.py --variants ./prompts/v1.txt --models gpt-4o,claude-3-5-sonnet --cases evals/cases.jsonl
    

    4. Projected Report Format

    ModelPass RateMedical SafetyProfessionalismLatency (p95)Cost/1k
    GPT-4o94%100%4.2/51.2s$0.015
    Claude 3.596%100%4.8/51.1s$0.012

    Next steps

    1. Add your API keys to the .env file to run the eval_runner.py.
    2. Review evals/results/ after the first run to calibrate the LLM judge.
    3. Import your last 5 production logs into cases.jsonl for better coverage.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Stop guessing if your prompt changes actually improved performance. This skill implements a systematic evaluation framework to quantify accuracy, latency, and cost across different models and prompt versions.

    What it does

    • Dataset generation builds a balanced collection of typical, edge, and adversarial test cases in .jsonl format.
    • Multi-layer grading utilizes code-based checks for schemas and regex alongside LLM-as-judge rubrics for qualitative analysis.
    • Benchmarking executes test runs across multiple models and prompt variants with concurrency and caching.
    • Regression tracking generates detailed reports comparing current performance against established baselines to flag score drops.
    • Error analysis isolates the worst-performing cases to identify specific failure modes in tool use, tone, or formatting.

    How it works

    1. Criteria definition establishes gradable metrics for correctness, safety, and format.
    2. Dataset construction populates evals/cases.jsonl with targeted inputs and expected outputs.
    3. Runner execution triggers a Python or TypeScript script to process cases across specified variants.
    4. Report generation produces tables with pass rates, p95 latency, and cost per variant.

    Frameworks & tools

    Uses Python or TypeScript for runners, JSONL for data storage, and LLM providers for judge-based evaluation.

    Why this beats prompting it yourself

    Manual spot-checking misses edge cases and fails to detect subtle regressions. This skill provides a reproducible, data-driven pipeline that treats prompt engineering like software engineering.

    Use cases

    • Compare the performance and cost of GPT-4o versus Claude 3.5 Sonnet for a specific task.
    • Verify that a safety update didn't break valid JSON output formatting.
    • Audit a new system prompt against a library of known production bugs.

    Known limitations

    Requires an existing LLM API key for grading and running tests. Human calibration is still recommended for the initial judge setup.

    Requires LLM API access for grading. Requires human calibration for the first 10% of cases to ensure judge accuracy.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    • Free with an Agensi account

    Permissions

    No special permissions declared or detected

    Listed4 days ago
    Updated1 day ago

    What's inside

    Frequently Asked Questions

    This harness automates the 'spot-checking' process by running your prompts through a library of test cases, using both code-based validation and AI-driven grading to provide a quantitative score for every change you make.

    The harness is framework-agnostic and supports any agent or model accessible via API, including OpenAI, Anthropic, and local models via Python or TypeScript runners.

    You will receive the complete evaluation framework including the dataset generator, automated grading scripts, regression reporting modules, and documentation for integrating it into your existing CI/CD pipeline.

    While it uses LLM-as-judge for qualitative analysis, it also includes 'cold' checks for JSON schema validation and regex matching to ensure your agent's output is technically sound without extra API costs.

    Setup involves defining your grading criteria and populating the JSONL test case file; the skill is designed to be 'plug-and-play' so you can start benchmarking your first variants within minutes.

    Yes, the regression tracking feature allows you to compare current test runs against a 'Gold Standard' baseline to immediately flag if a new prompt version caused a drop in accuracy or an increase in latency.

    New in Prompt & Skill Engineering