Test Failure Analyzer

The Problem

Your CI pipeline shows 15 red test failures. You open the log and see a wall of tracebacks from four different test files. Some are assertion errors ("expected 401, got 500"), some are environment errors ("Connection refused"), some are type errors, and some are timeouts. You spend 20 minutes reading through each one trying to figure out which failure causes which, which ones are related, and which to fix first. Then you discover that 12 of the 15 failures trace back to a single broken database fixture — and you could have fixed all 12 by fixing one thing.

Without structured analysis, test failures look like 15 independent problems. In reality, they're usually 3-4 root causes with cascading effects. Treating them as independent wastes time fixing symptoms instead of causes.

What You Get

• Multi-framework output parsing — pytest, unittest, Jest, Vitest, Go test, Mocha, RSpec, PHPUnit, JUnit, and Cargo test. Falls back to generic regex for unsupported frameworks
• Failure classification by type — Assertion (expected vs. actual mismatch), Environment (connection refused, missing service), Timeout (exceeded time limit), Type Error (wrong type passed), Fixture (shared test dependency broken), Syntax (code doesn't parse)
• Severity rating per failure — Critical (blocks all tests downstream), High (breaks a test module), Medium (isolated failure), Low (cosmetic, flaky, or skip-related)
• Cascading failure detection — groups failures by shared error messages and exception types to identify dependency chains: "DB connection failure → 4 downstream failures" with the root cause at the top
• Priority-ranked fix suggestions — concrete commands (not vague advice): "Start database server: docker compose up -d postgres", "Add error handling: wrap db calls in try/except", "Fix broken import in models.py line 12"
• Impact projection — estimated failures remaining after each fix: "#1 only → 2 remaining, #1 + #2 → 0 remaining"

Why This Beats Prompting It Yourself

Reading 15 test failures manually requires tracing each traceback, identifying the exception type, cross-referencing with other failures to find common roots, and prioritizing by which fix resolves the most failures. The analyzer automates this classification and pattern detection: it groups failures by exception type, builds a dependency graph to identify cascading chains, and produces a fix priority list where the first action resolves the most failures. What takes 20 minutes of manual triage becomes a 60-second report.

Example Session

You: "Here's the pytest output from our CI failure. 5 tests failed."

Agent:

FAILURE ANALYSIS REPORT
───────────────────────
Test Framework: pytest (Python)
Total Tests: 47 collected | Failed: 5 | Passed: 2
Failure Rate: 71% (Critical — suite is mostly broken)


FAILURE CLASSIFICATION
──────────────────────
#  Test                                 Type         Severity  Root Cause
─  ───────────────────────────────────  ───────────  ────────  ─────────────────────────
1  test_db.py::test_connection          Environment  Critical  DB server not running —
                                                               psycopg2.OperationalError
2  test_db.py::test_migration           Environment  Critical  Cascading from #1 — same
                                                               Connection refused
3  test_api.py::test_get_users          Environment  High      Cascading from #1 — API
                                                               can't reach DB
4  test_auth.py::test_login_invalid     Assertion    Medium    Server returns 500
                                                               instead of 401 —
                                                               unhandled DB exception
5  test_auth.py::test_login_rate_limit  Assertion    Medium    Returns 500 instead of
                                                               429 — same pattern as #4


CASCADING FAILURE
─────────────────

Use Cases

• CI triage — wake up to red CI and get a prioritized fix list in 60 seconds
• Post-refactor validation — after restructuring, quickly classify which failures are real regressions vs. outdated tests
• Sprint planning — use failure classification to estimate test debt and create tickets
• Release gate — analyze the full test suite before tagging a release to confirm all failures are understood
• Cross-service debugging — identify whether integration test failures are in your service, a dependency, or the test environment
• Onboarding — new team member sees red CI and gets an explanation of what's broken with fix steps

Known Limitations

The analyzer works on static test output — it cannot re-run tests automatically or interact with live test runners. It classifies failures by error patterns, not semantic understanding, so some root cause assignments may need manual confirmation. Stack trace depth is limited to 50 lines; deeply recursive failures may be truncated. It does not distinguish between code bugs and test bugs.