More screenshots
Works with the AI Agents you already use
Promptfoo Config Check
Check promptfoo eval and red team configs before they reach CI: written-in API keys, file:// paths that do not exist, keys sent over plain http, evals that cannot fail, unpinned graders, sharing on and duplicate tests.
$5
Promptfoo Config Check
Example session with this skill installed
Check this project with promptfoo-config-check.
- Read your context and instructions
- Compiled the promptfoo config check
$ python3 scripts/check_promptfoo.py examples/risky
BLOCK promptfooconfig.yaml:9 P02 MISSING_FILE_REFERENCE
- file://prompts/sytem.txt
Why: A file:// reference points to a file or glob that does not exist where promptfoo looks for it (the folder of the config that loads it, or of the test file it sits in), or exists only with different letter case, which fails on Linux and CI runners. promptfoo stops with an error for a missing prompt, provider, assertion, test or var file, and a test glob that matches nothing adds no tests, so the eval runs without them.
Safer: Fix the path relative to the config's folder (promptfoo resolves file:// paths from there), match the exact letter case, commit the file, and run promptfoo validate config in CI.
BLOCK promptfooconfig.yaml:14 P01 SECRET_IN_CONFIG
BLOCK promptfooconfig.yaml:19 P03 CREDENTIALS_OVER_PLAIN_HTTP
REVIEW promptfooconfig.yaml:6 P16 SECRET_OR_PERSONAL_DATA_IN_EVAL_DATA
REVIEW promptfooconfig.yaml:25 P15 CODE_FROM_OUTSIDE_PROJECT
REVIEW promptfooconfig.yaml:27 P14 RESULT_SHARING_ON
REVIEW tests/regression.yaml:6 P18 UNKNOWN_ASSERTION_TYPE
REVIEW tests/regression.yaml:8 P13 GRADER_NOT_PINNED
REVIEW tests/regression.yaml:11 P17 DUPLICATE_TEST
REVIEW tests/regression.yaml:17 P12 CHECK_ALWAYS_PASSES
REVIEW tests/regression.yaml:19 P11 NO_ASSERTIONS
REVIEW tests/regression.yaml:22 P16 SECRET_OR_PERSONAL_DATA_IN_EVAL_DATA
(Each finding also has Why and Safer lines, as in the first one.)
Verdict: BLOCK. Checked 2 files, 2 with promptfoo config content.
This is a static check of file text. NOT FLAGGED is not a guarantee.
Connects securely to your tools. The creator never sees your data.
About this skill
An apiKey pasted under a provider in promptfooconfig.yaml ends up in git and in every saved eval. A test glob such as file://tests/*.yml, when the files end in .yaml, adds no tests at all, and a suite whose tests have no assert passes in CI whatever the model answers. This skill finds those and the other promptfoo config mistakes below before you commit or run the eval, and tells you how to fix each one.
Your agent runs a scanner on promptfooconfig.yaml, .yml and .json files, other YAML or JSON files that are promptfoo configs by their content (such as a generated redteam.yaml), and the test, scenario, provider and defaultTest files a config loads with file://, globs included. It resolves file:// paths from the config's folder as promptfoo does, checks their exact letter case, reads only the header of CSV test files for __expected columns, and treats {{ env.NAME }} templates, apiKeyEnvar names, docs placeholders and keys for local servers as not secret. It never runs promptfoo, a provider, a script or an eval, and never contacts a model API. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach. Secrets are never printed.
What it checks
BLOCK
- P01: API keys, tokens, passwords and private keys written into a config or a file it loads: provider apiKey and other secret fields, env: entries, credential headers, auth blocks, TLS or signing keys, passwords in URLs, and values shaped like real provider keys anywhere, prompts and test vars included
- P02: file:// references to prompts, providers, test files or globs, var files, assertion scripts or transforms that do not exist, or exist only with different letter case
- P03: providers that send credentials to a remote host over plain http:// or ws://
REVIEW
- P11: evals with no assertions at all, and tests without assertions in a list where the others have them
- P12: checks that cannot fail: threshold 0 on minimum-score assertions or assert-sets, a test threshold of 0, weight 0 on every assertion, empty contains values, match-everything regexes, javascript or python checks that just return true
- P13: model-graded assertions (llm-rubric, factuality, g-eval, similar and others) with no pinned grader, and similar checks whose grader has no embedding model
- P14: result sharing turned on in the config (sharing: true, a sharing block, commandLineOptions.share), with the fix that actually stops uploads: sharing: false
- P15: provider, assertion, var or extension code loaded from an absolute path or from outside the project, and exec: providers that run a program by absolute path
- P16: secret-named test vars with literal values, email addresses, payment card numbers and US Social Security numbers in vars, and provider keys copied into env with {{ env.NAME }}
- P17: tests with the same description, or the same vars and assertions, in one list
- P18: misspelled assertion types (contain, llm_rubric, json) that promptfoo rejects as unknown
What's included
- The scanner (Python 3.9+, standard library only)
- A rules guide with the reason, safer approach and official promptfoo source for each rule
- A risky and a safe example eval (config, test file and prompt), and the real report the scanner produces on the risky one
Limits It reads YAML and JSON configs only: promptfooconfig.js and .ts files, command-line flags and values filled in from the environment at run time are not known. Files that a setup script downloads or a hook writes before the eval are reported as missing until they exist. Personal data detection covers email addresses, card numbers and US Social Security numbers only. Official examples without assertions or with the default grader get P11 or P13 by design. NOT FLAGGED is not a guarantee.
Static check of config text. JavaScript and TypeScript configs, run-time environment values and providers' live behavior are not checked; the assertion type list follows promptfoo's documentation as of November 2026. NOT FLAGGED is not a guarantee.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 4 days ago
Permissions
Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls. The example eval includes a clearly fake, inert key and password (NOTREAL-...) for the scanner to find. Nothing in it is run.