More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    llms.txt and AI Crawler Check

    1

    Check llms.txt and the AI crawler rules in robots.txt before you publish: missing H1, localhost or staging links, pages robots.txt blocks for every AI crawler, several bots on one User-agent line, misspelled bot names.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    llms.txt and AI Crawler Check

    llms.txt and AI Crawler Check

    Example session with this skill installed

    Check this project with llms-txt-ai-crawler-check.

    • Read your context and instructions
    • Compiled the llms txt

    $ python3 scripts/check_llms_txt.py examples/risky
    BLOCK docs-portal/static/llms.txt:1 L01 LLMS_TXT_WITHOUT_H1
    ## API v3 reference
    Why: llms.txt has no H1 title (# Name) before its first

    section. The llms.txt format makes the H1 with the site or project name its only required part, and tools that read the format start from it to find the summary and the link sections. A file that starts with a section, plain text or an HTML page is not an llms.txt file.

    Safer: Start the file with one H1 line naming the site or project (# Acme Docs), then an optional > summary line, then

    sections of - name: notes links. If the file is an HTML page, make the server return the plain Markdown file instead.

    BLOCK shop/public/llms.txt:9 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
    BLOCK shop/public/llms.txt:10 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
    BLOCK shop/public/llms.txt:11 L02 LOCAL_OR_STAGING_LINK
    BLOCK shop/public/llms.txt:12 L03 LLMS_CONTENT_BLOCKED_BY_ROBOTS
    BLOCK shop/public/llms.txt:16 L02 LOCAL_OR_STAGING_LINK
    BLOCK shop/public/llms.txt:22 L05 SECRET_IN_LLMS_LINK
    BLOCK shop/public/robots.txt:8 L04 USER_AGENT_LINE_NOT_ONE_NAME
    REVIEW docs-portal/static/llms.txt L18 LARGE_LLMS_FILE
    REVIEW shop/public/llms.txt:12 L16 DUPLICATE_LINK
    REVIEW shop/public/llms.txt:17 L17 HTTP_LINK
    REVIEW shop/public/llms.txt:18 L15 FILE_LIST_ITEM_INCOMPLETE
    REVIEW shop/public/robots.txt:11 L12 AI_AGENT_NAME_NOT_MATCHED
    REVIEW shop/public/robots.txt:14 L11 AI_SEARCH_AGENT_BLOCKED
    REVIEW shop/public/robots.txt:17 L13 AI_GROUP_DROPS_DEFAULT_RULES
    REVIEW shop/public/robots.txt:17 L14 USER_FETCHER_MAY_IGNORE_ROBOTS

    (Each finding also has Why and Safer lines, as in the first one.)
    Verdict: BLOCK. Checked 3 files, 3 with llms.txt or robots.txt content.
    This is a static check of file text. NOT FLAGGED is not a guarantee.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    User-agent: GPTBot, ClaudeBot looks like it blocks two AI crawlers, but Google's parser reads only the first name and stricter parsers match neither. An llms.txt that lists your /docs/.md pages while robots.txt says Disallow: /.md$ to everyone advertises pages that AI tools are told to skip, and a group that only says User-agent: ClaudeBot plus Crawl-delay: 1 quietly opens every path your * group closes. This skill finds those and the other llms.txt and AI crawler mistakes listed below before you publish, and tells you how to fix each one.

    Your agent runs a scanner on every llms.txt, llms-full.txt and robots.txt in your project's source folders (public/, static/, .well-known/ and the like). Build output folders are skipped in a project scan, so for an llms.txt that a Docusaurus, VitePress or Astro plugin generates, your agent builds the site and scans the output folder too. It reads llms.txt the way the llms.txt format describes it (H1 title, summary, sections of name: notes links), skipping code blocks and front matter, and reads robots.txt the way RFC 9309 describes it: one group per crawler, the longest matching rule wins, the * group only for crawlers no group names. When llms.txt and robots.txt sit in the same web folder (or Next.js app/robots.txt next to public/), it checks each llms.txt link against robots.txt for the AI crawler tokens of OpenAI, Anthropic, Google, Perplexity, Common Crawl, Apple, ByteDance and Meta. It never fetches your site. It returns BLOCK, REVIEW, or NOT FLAGGED, with the file, line, reason, and a safer approach, and each rule links to the llms.txt format, RFC 9309 or the crawler operator's page behind it. Secrets are never printed.

    What it checks

    BLOCK

    • L01: llms.txt without its H1 title (it starts with a section, plain text or an HTML page, or is empty)
    • L02: file-list links to localhost, private IP addresses, .local/.test/.internal names, staging hosts, Netlify deploy previews, Vercel branch URLs, ngrok or Cloudflare tunnels, or placeholder domains
    • L03: llms.txt links (or llms.txt itself) that robots.txt disallows for every known AI crawler and fetcher
    • L04: several AI crawler names on one User-agent line, or a full browser user-agent string
    • L05: links in llms.txt with a password, API key, token, JWT or signature in the URL, and real-looking passwords in database URLs in code examples

    REVIEW

    • L11: AI search and answer agents (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, meta-webindexer) blocked from the whole site while llms.txt is published
    • L12: misspelled, versioned or outdated AI crawler names (GPT-Bot, Google-Extend, GPTBot/1.1, ChatGPT, anthropic-ai without ClaudeBot)
    • L13: AI crawler groups that do not repeat the * group's Disallow rules, so those paths are open to them
    • L14: blocks on ChatGPT-User, Perplexity-User or meta-externalfetcher, which their operators say may not follow robots.txt
    • L15: list items without a link in an llms.txt section of links, or many links and none with a description
    • L16: the same URL listed twice in llms.txt
    • L17: http:// links in llms.txt
    • L18: llms.txt over 100,000 characters or llms-full.txt over 1 MB

    What's included

    • The scanner (Python 3.9+, standard library only)
    • A rules guide with the reason, safer approach and official source link for each rule
    • A risky and a safe example project (a shop and a docs portal), and the real report the scanner produces on the risky one

    Limits It reads the files as written. Build output folders (build/, dist/, .next/ and similar) are skipped when it scans a project; scan the output folder directly to check a generated llms.txt. robots.txt generated at build time (such as Next.js app/robots.ts), CDN or host settings and X-Robots-Tag headers are not read, and links are not fetched, so broken pages are not found. Whether a crawler obeys robots.txt is up to its operator, and new AI crawlers appear often. Classic search-engine rules in robots.txt are out of scope. NOT FLAGGED is not a guarantee.

    Static check of llms.txt, llms-full.txt and robots.txt text. Robots rules made at build time or by a CDN, the pages behind the links, and real crawler behavior are not checked. NOT FLAGGED is not a guarantee.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 4 days ago

    Permissions

    Terminal / Shell
    Read Files

    Allowed Hosts

    .

    Runs one local Python script in the terminal. It reads your project's files, writes nothing, and makes no network calls. The example project includes a clearly fake key (NOTREAL-EXAMPLE-KEY) in one llms.txt link for the scanner to find, and fake example.com hosts. Nothing in it is run or fetched.

    Listed4 days ago

    What's inside

    Frequently Asked Questions

    No. The script reads local files only and makes no network calls.

    No. It only reports. Your agent changes files only if you ask it to.

    Any agent that supports SKILL.md skills and can run a terminal command, such as Claude Code. It needs Python 3.9 or newer and nothing else.

    No. It reads the files in your project or build output folder. It never fetches a URL or contacts a crawler, and secrets in links are masked.

    BLOCK: fix it before you ship. REVIEW: check it before release. NOT FLAGGED: no rule matched. It is a static check, so NOT FLAGGED is not a guarantee.

    The scanner, a rules guide with the reason and safer approach for every rule, a risky and a safe example llms.txt with robots.txt, and the real report the scanner prints on the risky example.

    New in Marketing & SEO