Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    incident derived agent evaluation dataset

    1

    Construct provenance-linked, deduplicated evaluation datasets from real-world agent incident logs.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    incident derived agent evaluation dataset

    incident derived agent evaluation dataset

    Example session with this skill installed

    Build an evaluation dataset from the three redacted API timeout incidents in incident-logs/q3-failures.json. Deduplicate and split by incident family.

    • Read your context and instructions
    • Compiled the incident derived agent

    I have processed the incident logs, deduplicated the failure traces, and generated a provenance-linked dataset manifest. The incidents were grouped into the 'Connection Timeout' family and split to preserve a held-out set.

    Incident-Derived Dataset Manifest

    MetricValue
    Total Incidents Processed3
    Unique Prompt Families1
    Deduplicated Records2
    Source Provenanceincident-logs/q3-failures.json
    Split Strategy50% Production / 50% Held-out
    Artifact Path./outputs/dataset_2026_09_18.json

    Dataset Content Summary

    [
      {
        "id": "CASE-001",
        "incident_family": "api_timeout_retry_exhaustion",
        "minimal_context": "User requests high-latency reporting endpoint; auth token expires during 30s wait.",
        "expected_behavior": "Agent should refresh token or return 408 status; current failure shows 500 loop.",
        "provenance_ref": "q3-failures.json:line_45",
        "tags": ["network", "auth-lifecycle"]
      },
      {
        "id": "CASE-002",
        "incident_family": "api_timeout_retry_exhaustion",
        "minimal_context": "Parallel execution of 5 data fetch calls exceeds concurrent connection limit.",
        "expected_behavior": "Queue management or exponential backoff implementation.",
        "provenance_ref": "q3-failures.json:line_112",
        "tags": ["concurrency", "rate-limiting"]
      }
    ]
    

    Next steps

    1. Run scripts/build_dataset.py to export the deterministic JSONL for your evaluation pipeline.
    2. Review the semantic duplicates in ./outputs/review_queue.md to confirm the deduplication logic.
    3. Apply the held-out set to your CI/CD gate for the next agent release.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    General LLM benchmarks fail to capture how your specific agent breaks in production. Generic evaluation datasets lack the provenance needed to trace failures back to real-world incident root causes.

    What it does

    • Extracts minimal reproducible prompts from redacted incident logs and failure traces.
    • Groups paraphrases and near-duplicate incidents to prevent data leakage across splits.
    • Separates verified facts from proposed explanations to ensure objective ground truth.
    • Generates a deterministic dataset manifest with linked source provenance and versioning.
    • Identifies label conflicts and semantic duplicates for manual review.

    Frameworks & tools

    Python scripts (build_dataset.py), JSON schema for incident fixtures, and provenance-linked reporting.

    Why this beats prompting it yourself

    Manual dataset creation often leads to data contamination where similar prompts appear in both train and test sets. This skill enforces strict evidence discipline, ensuring every evaluation case is linked to a real incident without leaking sensitive credentials or model-generated hallucinations.

    Use cases

    • Building a gold-standard evaluation set for agentic workflows.
    • Deduplicating incident reports to create clean fine-tuning data.
    • Auditing agent performance against historical production failures.
    • Preparing held-out test sets for rigorous model deployment gates.

    Known limitations

    Requires pre-redacted incident cases as input. Does not perform model evaluation itself or invent ground truth where source data is missing.

    Requires pre-redacted incident inputs. Does not perform actual model inference or provide accuracy metrics. Coverage is limited to supplied incident families.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    Requires

    Python

    Permissions

    No special permissions declared or detected

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    This skill creates high-fidelity evaluation datasets by extracting and deduplicating prompts from your real-world incident logs, ensuring your testing reflects actual production failures rather than generic benchmarks.

    It is framework-agnostic and works with any agent architecture that logs interactions, as long as the inputs are provided in a redacted JSON format.

    The tool includes specialized grouping logic to identify paraphrased or near-duplicate incidents, preventing similar cases from appearing in both your training and testing sets to ensure valid results.

    You receive a suite of Python scripts for dataset construction, the JSON schema for incident fixtures, and documentation on generating provenance-linked reports.

    No, this skill requires your incident logs to be pre-redacted; it focuses on structuring, deduplicating, and verifying the evidence rather than stripping sensitive PII.

    Each dataset generated includes a deterministic manifest that links every evaluation case back to its original source incident, allowing for full auditability and version control as your agent evolves.

    New in LLM Ops