- Home
- Skills
- Testing & Debugging
- Test Tamper Detector: Catch Agents Gaming Your Tests
Test Tamper Detector: Catch Agents Gaming Your Tests
**The pain.** You ask a coding agent to make the failing tests pass.
$12
Works with the AI Agents you already use
Test Tamper Detector: Catch Agents Gaming Your Tests
Example session with this skill installed
My coding agent says it fixed the pricing bugs and "all tests pass". Here is its diff. Did it actually fix the code, or did it change the tests?
- Read your context and instructions
- Compiled the test tamper detector
Test Tamper Report
Input: samples/agent-change.diff.txt (DEMO)
Files: 7 (test 4, source 1, config/CI 2, snapshot 0)
Findings: 11 high, 5 medium, 0 low, 0 accepted by a human
Verdict: TAMPER_LIKELY - at least one change can make failing tests pass without fixing the code. Review before merging.
| # | Severity | Rule | Location | Evidence |
|---|---|---|---|---|
| 1 | HIGH | CI_FAILURE_MASKED | .github/workflows/ci.yml:10 | continue-on-error: true |
| 2 | HIGH | TESTS_EXCLUDED_IN_CONFIG | pyproject.toml:2 | addopts = "-q --deselect tests/test_pricing.py::test_bulk_price_threshold" |
| 3 | HIGH | COVERAGE_THRESHOLD_LOWERED | pyproject.toml:5 | fail_under = 70 <br>fail_under: 90 -> 70 |
| 4 | HIGH | SPECIAL_CASE_MATCHES_TEST | src/pricing.py:8 | if total == 19.99 and percent == 15: <br>input 19.99 and output 16.99 appear together in an existing test |
| 5 | HIGH | SKIP_ADDED | tests/test_pricing.py:9 | @pytest.mark.skip(reason="flaky on CI") |
| 6 | HIGH | TEST_REMOVED | tests/test_pricing.py:14 (old) | def test_discount_full_price(): <br>test 'test_discount_full_price' removed |
| 7 | HIGH | ASSERTION_WEAKENED | tests/test_pricing.py:22 | assert result <br>was: assert result == 334.80 |
| 8 | HIGH | TEST_FILE_DELETED | tests/test_tax.py:1 (old) | deleted file tests/test_tax.py <br>2 test(s) removed with the file: test_tax_after_discount, test_tax_zero_rate |
| 9 | HIGH | FOCUS_ADDED | web/cart.test.ts:5 | it.only("sums line items", () => { |
| 10 | HIGH | ASSERTION_WEAKENED | web/cart.test.ts:11 | expect(cartTotal([{ price: 10, qty: 1 }], "SAVE10")).toBeTruthy(); <br>was: expect(cartTotal([{ price: 10, qty: 1 }], "SAVE10")).toEqual(9); |
| 11 | HIGH | EXCEPTION_SWALLOWED_IN_TEST | web/cart.test.ts:12 | } catch (e) {} |
| 12 | MEDIUM | HARDCODED_RETURN_MATCHES_TEST | src/pricing.py:9 | return 16.99 <br>tests use: 16.99 |
| 13 | MEDIUM | COVERAGE_EXCLUSION_ADDED | src/pricing.py:20 | def _debug_dump(rows): # pragma: no cover (tamper-ok: local debugging helper, never shipped) <br>author note (not counted as acceptance): local debugging helper, never shipped |
| 14 | MEDIUM | ASSERTIONS_DROPPED | tests/test_pricing.py:15 (old) | assert apply_discount(149.99, 0) == 149.99 <br>6 assertion line(s) removed, 4 added across test files (net -2) |
| 15 | MEDIUM | TOLERANCE_LOOSENED | tests/test_pricing.py:16 | assert bulk_price(2.50, 50) == pytest.approx(112.50, rel=0.05) <br>explicit tolerance added to approx() |
| 16 | MEDIUM | EXPECTED_VALUE_CHANGED | tests/test_pricing.py:17 | assert bulk_price(2.50, 49) == 122.25 <br>was: assert bulk_price(2.50, 49) == 122.50 |
Paste this to the agent
Before this change can merge, restore each item below.
Do not edit, skip or weaken tests to make them pass, and do not add notes that mark items as approved.
If you believe an item is a real requirement change, reply with the ticket or spec that says so; a human decides.
1. .github/workflows/ci.yml:10 [CI_FAILURE_MASKED] Remove the mask so a failing step fails the build.
2. pyproject.toml:2 [TESTS_EXCLUDED_IN_CONFIG] Which tests stop running, and why?
3. pyproject.toml:5 [COVERAGE_THRESHOLD_LOWERED] Restore the threshold and add tests instead.
4. src/pricing.py:8 [SPECIAL_CASE_MATCHES_TEST] Is this a general rule, or a hard-coded answer for the test input?
5. tests/test_pricing.py:9 [SKIP_ADDED] Remove the skip and fix the failure, or link the issue that tracks it.
...
Connects securely to your tools. The creator never sees your data.
About this skill
The pain. You ask a coding agent to make the failing tests pass. It comes back with "all tests pass". Sometimes it fixed the code. Sometimes it added .skip, turned toEqual(9) into toBeTruthy(), deleted the test file, set continue-on-error: true, dropped the coverage gate from 90 to 70, or wrote if total == 19.99: return 16.99 so one test goes green. Both versions look the same in the agent's summary, and the second kind is easy to miss in a long diff.
The outcome. A verdict on the diff (TAMPER_LIKELY, REVIEW or CLEAN), every suspicious change with file:line, what the line was before, why it matters, and a numbered "paste this to the agent" block that tells it exactly what to restore. Exit code 1 on TAMPER_LIKELY, so it also works as a CI or pre-commit gate.
The mechanism. A Python script (standard library only, no network, never edits your files) parses the unified diff and runs 37 rules across pytest/unittest, Jest/Vitest/Mocha/Playwright, Go/testify, JUnit/AssertJ, xUnit/NUnit/MSTest and RSpec, plus runner, coverage, build and CI configs and .gitattributes. It pairs each removed assertion with its replacement and ranks their strength (exact value > shape > truthiness). With --git it also searches the test files at the base revision, so a hard-coded answer in source is caught even when no test file changed. Notes written inside the diff (tamper-ok: ...) are shown but never accepted; only a human accepts a finding, on the command line. The agent then judges each finding as confirmed, legitimate or unclear.
The limits. It reads the diff, not your intent: a real requirement change triggers the same patterns, which is why findings are questions and acceptance is explicit. It does not run tests, prove they pass, find bugs, or judge whether the original tests were good. Small integers (if x == 7) are not matched, to keep noise down. Works better with Git (reads the diff and base tests itself) and the GitHub CLI (gh pr diff); pairs with a run receipt such as Proof-of-Done Receipts, which proves the tests ran.
Reads the diff, not intent: a legitimate requirement change triggers the same patterns, so each finding needs a human or agent judgment and acceptance is explicit (--accept / --allow). Does not run tests or check that they pass, does not find bugs, and does not judge whether the original tests were good. Assertion pairing is line-based; a large test-file rewrite shows up as dropped or renamed tests and needs a human read. Hard-coded-answer checks ignore small integers (if x == 7) and word-only values give at most MEDIUM. Skips applied by command-line flags or plugins outside the repo are not seen. Dependency manifests and lockfiles are skipped.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 5 days ago
Requires
Permissions
File Scopes
No network calls. Never edits, reverts or fixes your code or tests.