More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Promptfoo Config Check

    1

    Check promptfoo eval and red team configs before they reach CI: written-in API keys, file:// paths that do not exist, keys sent over plain http, evals that cannot fail, unpinned graders, sharing on and duplicate tests.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Promptfoo Config Check

    Promptfoo Config Check

    Example session with this skill installed

    Check this project with promptfoo-config-check.

    • Read your context and instructions
    • Compiled the promptfoo config check

    $ python3 scripts/check_promptfoo.py examples/risky
    BLOCK promptfooconfig.yaml:9 P02 MISSING_FILE_REFERENCE
    - file://prompts/sytem.txt
    Why: A file:// reference points to a file or glob that does not exist where promptfoo looks for it (the folder of the config that loads it, or of the test file it sits in), or exists only with different letter case, which fails on Linux and CI runners. promptfoo stops with an error for a missing prompt, provider, assertion, test or var file, and a test glob that matches nothing adds no tests, so the eval runs without them.
    Safer: Fix the path relative to the config's folder (promptfoo resolves file:// paths from there), match the exact letter case, commit the file, and run promptfoo validate config in CI.

    BLOCK promptfooconfig.yaml:14 P01 SECRET_IN_CONFIG
    BLOCK promptfooconfig.yaml:19 P03 CREDENTIALS_OVER_PLAIN_HTTP
    REVIEW promptfooconfig.yaml:6 P16 SECRET_OR_PERSONAL_DATA_IN_EVAL_DATA
    REVIEW promptfooconfig.yaml:25 P15 CODE_FROM_OUTSIDE_PROJECT
    REVIEW promptfooconfig.yaml:27 P14 RESULT_SHARING_ON
    REVIEW tests/regression.yaml:6 P18 UNKNOWN_ASSERTION_TYPE
    REVIEW tests/regression.yaml:8 P13 GRADER_NOT_PINNED
    REVIEW tests/regression.yaml:11 P17 DUPLICATE_TEST
    REVIEW tests/regression.yaml:17 P12 CHECK_ALWAYS_PASSES
    REVIEW tests/regression.yaml:19 P11 NO_ASSERTIONS
    REVIEW tests/regression.yaml:22 P16 SECRET_OR_PERSONAL_DATA_IN_EVAL_DATA

    (Each finding also has Why and Safer lines, as in the first one.)
    Verdict: BLOCK. Checked 2 files, 2 with promptfoo config content.
    This is a static check of file text. NOT FLAGGED is not a guarantee.

    BLOCK promptfooconfig.yaml

    Connects securely to your tools. The creator never sees your data.

    About this skill

    An apiKey pasted under a provider in promptfooconfig.yaml ends up in git and in every saved eval. A test glob such as file://tests/*.yml, when the files end in .yaml, adds no tests at all, and a suite whose tests have no assert passes in CI whatever the model answers. This skill finds those and the other promptfoo config mistakes below before you commit or run the eval, and tells you how to fix each one.

    Your agent runs a scanner on promptfooconfig.yaml, .yml and .json files, other YAML or JSON files that are promptfoo configs by their content (such as a generated redteam.yaml), and the test, scenario, provider and defaultTest files a config loads with file://, globs included. It resolves file:// paths from the config's folder as promptfoo does, checks their exact letter case, reads only the header of CSV test files for __expected columns, and treats {{ env.NAME }} templates, apiKeyEnvar names, docs placeholders and keys for local servers as not secret. It never runs promptfoo, a provider, a script or an eval, and never contacts a model API. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach. Secrets are never printed.

    What it checks

    BLOCK

    • P01: API keys, tokens, passwords and private keys written into a config or a file it loads: provider apiKey and other secret fields, env: entries, credential headers, auth blocks, TLS or signing keys, passwords in URLs, and values shaped like real provider keys anywhere, prompts and test vars included
    • P02: file:// references to prompts, providers, test files or globs, var files, assertion scripts or transforms that do not exist, or exist only with different letter case
    • P03: providers that send credentials to a remote host over plain http:// or ws://

    REVIEW

    • P11: evals with no assertions at all, and tests without assertions in a list where the others have them
    • P12: checks that cannot fail: threshold 0 on minimum-score assertions or assert-sets, a test threshold of 0, weight 0 on every assertion, empty contains values, match-everything regexes, javascript or python checks that just return true
    • P13: model-graded assertions (llm-rubric, factuality, g-eval, similar and others) with no pinned grader, and similar checks whose grader has no embedding model
    • P14: result sharing turned on in the config (sharing: true, a sharing block, commandLineOptions.share), with the fix that actually stops uploads: sharing: false
    • P15: provider, assertion, var or extension code loaded from an absolute path or from outside the project, and exec: providers that run a program by absolute path
    • P16: secret-named test vars with literal values, email addresses, payment card numbers and US Social Security numbers in vars, and provider keys copied into env with {{ env.NAME }}
    • P17: tests with the same description, or the same vars and assertions, in one list
    • P18: misspelled assertion types (contain, llm_rubric, json) that promptfoo rejects as unknown

    What's included

    • The scanner (Python 3.9+, standard library only)
    • A rules guide with the reason, safer approach and official promptfoo source for each rule
    • A risky and a safe example eval (config, test file and prompt), and the real report the scanner produces on the risky one

    Limits It reads YAML and JSON configs only: promptfooconfig.js and .ts files, command-line flags and values filled in from the environment at run time are not known. Files that a setup script downloads or a hook writes before the eval are reported as missing until they exist. Personal data detection covers email addresses, card numbers and US Social Security numbers only. Official examples without assertions or with the default grader get P11 or P13 by design. NOT FLAGGED is not a guarantee.

    Static check of config text. JavaScript and TypeScript configs, run-time environment values and providers' live behavior are not checked; the assertion type list follows promptfoo's documentation as of November 2026. NOT FLAGGED is not a guarantee.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    Permissions

    Terminal / Shell
    Read Files

    Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls. The example eval includes a clearly fake, inert key and password (NOTREAL-...) for the scanner to find. Nothing in it is run.

    Listed4 days ago

    What's inside

    Frequently Asked Questions

    No. The script reads local files only and makes no network calls.

    No. It only reports. Your agent changes files only if you ask it to.

    Any agent that supports SKILL.md skills and can run a terminal command, such as Claude Code. It needs Python 3.9 or newer and nothing else.

    No. It reads promptfoo configs and the files they load as text. It never runs promptfoo, never calls a model provider, and keys it finds are masked.

    BLOCK: fix it before you ship. REVIEW: check it before release. NOT FLAGGED: no rule matched. It is a static check, so NOT FLAGGED is not a guarantee.

    The scanner, a rules guide with the reason and safer approach for every rule, a risky and a safe example promptfoo eval setup, and the real report the scanner prints on the risky example.

    New in LLM Ops