prompt dataset tester

The problem

Developers often find prompt vulnerabilities only after users break them in production. Manual testing is slow and misses adversarial edge cases like injection or instruction leakage.

What it does

• Generates a failure-mode map covering instruction conflicts, format breaks, and context overflows.
• Produces runnable adversarial test packs with hidden injection payloads and role-confusion inputs.
• Tests boundary conditions including Unicode tricks, delimiter collisions, and empty inputs.
• Delivers pass/fail reports with specific prompt patches and regression risk assessments.

Why this beats prompting it yourself

This skill enforces a rigorous QA methodology that separates facts from hypotheses. It systematicially probes for weaknesses that standard LLMs tend to overlook when asked to "check for bugs" in their own instructions.

Use cases

• Red-teaming a new system prompt before deploying to a production RAG application.
• Generating a JSONL dataset of edge cases for automated CI/CD prompt evaluation.
• Hardening prompts against prompt injection via user-supplied data inputs.

Known limitations

Security conclusions are labeled as hypotheses until verified by live model execution. Model-specific behavior requires the user to define the target LLM.