Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Agent Tool-Trace Audit

    1

    This skill reads the run itself (the input, every tool call and the final output) and checks it against rules you wrote down in advance.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Agent Tool-Trace Audit

    Agent Tool-Trace Audit

    Example session with this skill installed

    Our refund agent handled ticket run-0413 overnight and I've got a bad feeling about it. Here's the trace as trace.jsonl and our policy.json. Did it stay inside the rules?

    • Read your context and instructions
    • Compiled the agent tool-trace audit

    I ran agent-tool-trace-audit check on trace.jsonl against policy.json (refund-agent-v1, 7 rules). Verdict: FAIL (exit code 1), five violations.

    {
      "kind": "agent-tool-trace-audit",
      "agent": "refund-bot",
      "runId": "run-0413",
      "policy": "refund-agent-v1",
      "steps": 4,
      "verdict": "FAIL",
      "rulesChecked": 7,
      "violations": [
        { "rule": "allowed-tools", "step": 2, "why": "called a tool not in the allow-list: run_shell" },
        { "rule": "forbidden-tools", "step": 2, "why": "called a forbidden tool: run_shell" },
        { "rule": "must-call", "step": -1, "why": "never called the required tool: lookup_order" },
        { "rule": "invariant", "step": 3, "why": "invariant failed: amount lte 100 (was 250)" },
        { "rule": "no-content", "step": 2, "why": "content contained a forbidden substring" }
      ],
      "traceSha256": "b9fca1c94c0bd209bbbb5b668c5c032f7f315bcb4744628866e69fcd8d72dfc5",
      "policySha256": "9d0ffa7181875209bd88352aab86c6f0bc59852b63fe0d0bb9d4da8368de6a48",
      "reportSha256": "4d70ba96010b4fd4943b4b22502a377e4252dc419e9b488ebd02ded513645354"
    }
    

    What happened

    1. At step 2 the agent ran a shell command (run_shell), which the policy forbids, and read a .env file.
    2. That same step put a live payment key prefix (sk_live_) into the trace, so treat the key as exposed.
    3. It issued a refund without ever calling lookup_order, so nothing checked the order first.
    4. The refund was 250, over the policy's limit of 100.
      I'd rotate that key now, reverse or review refund R-5532, and block run_shell at the tool layer before this agent runs again.
    agent-tool-trace-audit check on trace.jsonl against policy.json

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Agent Trust: check what the agent did, not what it says it did

    Agents report success in prose. This skill reads the run itself (the input, every tool call and the final output) and checks it against rules you wrote down in advance. No model grades another model: every rule is a plain deterministic check, and every violation names the step it happened on.

    What it does

    Given a trace and a policy, it checks:

    • allowed-tools / forbidden-tools: every tool the agent called
    • must-call: a tool that has to appear (for example a lookup before a refund)
    • output-json: the final output contains a JSON object with the required fields
    • invariant: a numeric or exact limit on an output field (amount <= 100, status == "ok")
    • no-content: a string that must never appear in any step (key prefixes, internal hostnames)
    • max-steps: a ceiling on run length

    It also runs a regression suite (suite): a fixed set of scenarios must all pass, on at least three, to be certified. The result is a hashed receipt that says PASSES or FAILS.

    When an agent should use it

    • Reviewing a run that went wrong, or one you are about to trust
    • Before shipping a prompt, tool or model change
    • Keeping an audit trail of agent runs that shows if anyone edits it later

    What it outputs

    JSON: PASS/FAIL, rules checked, every violation (rule, step, reason), and SHA-256 hashes of the trace, policy and report. With --ledger, each check is appended to a hash-chained ledger; verify-ledger names the first entry that was altered. Exit 0 = PASS, 1 = FAIL, 2 = bad input.

    Limitations

    • It proves the run obeyed your written rules. It does not judge whether the answer was right or well written.
    • Your logs need converting to the step format (input, tool, output) first. JSONL with one step per line is accepted.
    • output-json and invariant read the first JSON object in the final output step.

    Open source, packaged skill

    Original work by Simon Gant. The core logic is public under the MIT licence (agent-proof). This listing is the packaged, tested agent skill: SKILL.md workflow, CLI, JSONL support, ledger, suite receipt and docs. Requires Node.js 18+, zero dependencies, no network, no API keys.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed today

    Requires

    Node.js 18+
    Node.js

    Permissions

    Runs one local Node.js script (`node scripts/agent-tool-trace-audit.js`) through the agent's shell tool. Reads only the trace, policy and results files you pass it. Writes a file only when you pass `--ledger` (the audit ledger you name). No network access, no environment variables read. Pure compute.

    Listedtoday

    What's inside

    Frequently Asked Questions

    No. Every rule is a deterministic check, so the same trace and policy always give the same verdict and hash.

    JSON `{agent, runId, steps:[{i, kind, content, tool}]}` or JSONL with one step per line. Most agent frameworks' logs convert in a few lines.

    `check` audits one run. `suite` runs a fixed set of scenarios against the same policy and certifies the agent only if all pass, on at least three.

    They can edit the file, but `verify-ledger` will show the first entry that no longer matches its hash chain.

    The core logic is MIT open source. You are paying for the packaged, tested agent skill and workflow.

    New in LLM Ops