LLM safety-path audit

    1

    A step-by-step audit checklist for the safety path of an LLM feature (input judge, decision, output gate, background tasks, keys, logs), with a report template.

    Free

    Security scanned

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    LLM safety-path audit

    Example session with this skill installed

    Audit the chat handler in src/api/chat.ts for fail-closed logic and log safety. Review the sanitize() and gatekeeper() functions specifically.

    • Read your context and instructions
    • Compiled the llm safety-path audit

    Request

    ▎ Audit the safety path of this chat app: the judge, the decision function, the output
    ▎ gate and the background tasks.

    Result

    ▎ Summary: 9 issues: 2 high, 3 medium, 4 low. All fixed, each with a test that fails
    ▎ without the fix. The design failed closed. The weakest remaining point is that the
    ▎ confidence thresholds were never tuned on real conversations.
    ▎
    ▎ | # | Severity | Finding
    ▎ | Fix
    ▎ |
    ▎ |-----|----------|--------------------------------------------------------------------
    ▎ ----------------------------------------------|---------------------------------------
    ▎ -------------------------------------------------|
    ▎ | 1 | High | The output gate read only the first 12,000 characters, so a long
    ▎ reply could pass with its second half unchecked | Check long drafts in overlapping
    ▎ pieces; any flagged piece flags the whole draft |
    ▎ | 2 | High | The judge could lose a clear verdict when its quotation was longer
    ▎ than the stored evidence | Match the whole quotation; shorten
    ▎ only the kept copy |
    ▎ | 3 | Medium | A blank reply showed as an empty turn, and a cut-off reply showed
    ▎ mid-sentence | Blank becomes "cannot answer"; a
    ▎ cut-off reply ends at its last whole sentence |
    ▎ | 5 | Medium | Background work held the conversation lock for up to three model
    ▎ calls | Give the background work its own
    ▎ lock, with a test that the next message does not wait |
    ▎
    ▎ (Findings 4 and 6–9 omitted here.)
    ▎

      ▎ Next steps: (1) fix findings 1 and 2 first, since they let a protection be bypassed; 
      ▎ (2) re-run the tests; (3) tune the thresholds on a larger labelled set and repeat the 
    

    ▎ review after any model change.
    ▎
    ▎ This is a sample from a review of my own project, not client work.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Safety code around an LLM often fails in quiet ways: an error lets a message through, a long reply is only partly checked, or a background task works on the wrong message. This skill walks an AI agent through an audit of that code and produces a ranked report.

    What it does

    • Fail-closed check: finds paths where an error, timeout or blank verdict lets content through.
    • Whole-text check: finds output gates that read only the first part of a long reply.
    • Background tasks: finds tasks that use the wrong message or hold the lock the next message needs.
    • Keys and logs: finds API keys or user text in logs, errors or API answers.
    • Clean-up: finds per-conversation maps and locks that are never dropped.

    How it works

    1. Find the safety path: judge, decision, reply, output gate, storage, background tasks.
    2. Go through the checklist and record file and line as evidence.
    3. Rate each finding High, Medium or Low.
    4. Write the report from the template, with a fix and a regression test for each finding.

    Works with Any language. It reads the code around your model calls, not the model. Based on A review I did of my own chat app's safety path: nine issues found, fixed and tested. Known limitations It reviews the code it is given. It cannot certify model behaviour or provider-side safety, and it does not test live against paid APIs. A review is a snapshot, not a guarantee.

    Audits are restricted to provided source code. Cannot verify provider-side safety or UI-level vulnerabilities. Requires manual confirmation of thresholds not tuned on real data.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 6 days ago

    • Free with an Agensi account

    Permissions

    Read Files
    Terminal / Shell
    Write Files
    Listed6 days ago

    What's inside

    Frequently Asked Questions

    This skill identifies structural code vulnerabilities—such as 'fail-open' logic during timeouts or partial output filtering—that standard prompts and automated scanners often miss.

    You will receive a comprehensive vulnerability report with High, Medium, and Low risk rankings, along with specific remediation steps and regression tests for every issue found.

    Yes, this skill is framework-agnostic and compatible with any LLM backend built in Python, Node.js, or Go, including popular tools like LangChain and the OpenAI SDK.

    This skill audits your application-level logic and integration code; it does not perform live penetration testing or interact directly with your paid API endpoints.

    Simply provide the source code or specific files responsible for your input/output gates and message processing logic to begin the audit process.

    It ensures that if your safety judge fails or times out, the system defaults to a 'blocked' state rather than accidentally allowing unvetted content through to the user.

    New in Security