- Home
- Skills
- Testing & Debugging
- Agent Failure Regression Set Builder: Evals From Real Bugs
Works with the AI Agents you already use
Agent Failure Regression Set Builder: Evals From Real Bugs
Turns logged agent failures into a deduplicated regression set with pass rules, a JSONL dataset, a runner script and a pre-deploy gate.
$9.99
Agent Failure Regression Set Builder: Evals From Real Bugs
Example session with this skill installed
Turn these logs into a regression set. The agent failed to call the 'get_weather' tool twice when the user asked for London, and once it gave the temperature in Celsius instead of Fahrenheit.
- Read your context and instructions
- Compiled the agent failure regression
I have analyzed the provided logs and clustered the failures into two categories: Tool Calling and Formatting. I also added two "should still work" cases to ensure basic query handling remains stable.
Failure Analysis and Regression Set
FAILURE CLUSTERS
| Cause | Count | Example |
|---|---|---|
| Missing Tool Call | 2 | User asked for London weather; agent replied with text only. |
| Unit Mismatch | 1 | Expected Fahrenheit but agent returned Celsius. |
REGRESSION SET: regressions.jsonl
{"id": "tc_001", "cluster": "tool_calling", "source": "log_line_42", "input": "What is the weather in London?", "context": null, "expected": "Call get_weather with location=London", "check": "tool_call", "tool_name": "get_weather", "args": {"location": "London"}, "severity": "BLOCKER"}
{"id": "fmt_001", "cluster": "formatting", "source": "log_line_89", "input": "Get weather for NYC in Fahrenheit", "context": "weather_data: 22C", "expected": "Convert 22C to 72F", "check": "contains", "value": "72", "severity": "BLOCKER"}
{"id": "pos_001", "cluster": "baseline", "source": "added", "input": "Hello", "context": null, "expected": "Helpful greeting", "check": "rubric", "rubric": "Does the agent greet the user without calling tools?", "severity": "WATCH"}
RUNNER: run_regressions.py
import json
def run_agent(user_input, context):
# TODO: Connect your agent here
# return agent_output
pass
def check_result(case, output):
if case["check"] == "tool_call":
# Logic to verify tool name and args
return True
if case["check"] == "contains":
return case["value"] in str(output)
return False
def main():
with open("regressions.jsonl", "r") as f:
cases = [json.loads(line) for line in f]
results = []
for case in cases:
out = run_agent(case["input"], case["context"])
passed = check_result(case, out)
results.append((case["id"], case["severity"], passed))
# Print summary table and exit(1) if any BLOCKER fails
print("Test Summary Complete.")
if __name__ == "__main__":
main()
GATE
- CI Command:
python run_regressions.py - Failure Rule: Any
BLOCKERfailure triggers a non-zero exit code.
Next steps
- Implement the
run_agentfunction inrun_regressions.pyto call your LLM or agent class. - Add the runner to your pre-commit hook or GitHub Action.
- Add a new JSONL line for every future production bug reported.
Connects securely to your tools. The creator never sees your data.
About this skill
Generic eval suites test what the builder imagined. Real agents fail in specific, repeatable ways, and the same bug returns after the next prompt tweak or model swap. This skill builds the test set from failures that already happened.
What it does
- Normalizes failures: turns logs, traces, transcripts and tickets into input, actual behavior, expected behavior and source.
- Clusters by root cause: ignored instruction, wrong tool or arguments, made-up facts, wrong refusals, broken format, lost context, unsafe actions and more, with counts.
- Dedupes and writes cases: 1 to 3 clear cases per pattern plus a paraphrase, each with a pass rule (exact, contains, regex, JSON schema, tool-call or a short rubric).
- Guards against over-correction: adds should-still-work cases so a fix that refuses everything is caught.
- Runner and gate: a JSONL dataset, a small runner script and a BLOCKER / WATCH gate that fails CI when a blocker regresses.
How it works
- Paste failure examples in any format and describe what the agent should do.
- Review the clusters and cases.
- Connect the runner to your agent with one function and add it to your deploy step.
Builds cases only from failures you provide. The runner is written, not run against your agent; rubric checks need a judge model you connect. Remove personal data and secrets before sharing logs.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 12 days ago