Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    llms.txt + AI Crawler Access Skill Pack

    1

    Audit whether AI crawlers can reach your key pages and draft a practical llms.txt — without cloaking, inventing URLs, or promising ranking gains.

    $29

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    llms.txt + AI Crawler Access Skill Pack

    llms.txt + AI Crawler Access Skill Pack

    Example session with this skill installed

    Audit AI crawler access for https://example.com focusing on /guides/* and /docs/*. Draft an llms.txt that only lists URLs that exist. Recommend robots changes; don't apply them.

    • Read your context and instructions
    • Compiled the llms txt

    EXAMPLE — fictional site, illustrative output only.

    EXAMPLE audit: /guides/ allowed; /docs/ disallowed to unknown bots via broad Disallow: /docs — may block AI crawlers unintentionally.

    EXAMPLE llms.txt draft: # EXAMPLE title + 5 real guide URLs + contact.

    EXAMPLE robots note: Narrow the /docs rule; document GPTBot/ClaudeBot intent in the matrix. Publishing is a human step.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    llms.txt + AI Crawler Access is a Claude / Cursor skill for the technical access layer of GEO/AEO: robots policy, AI bot allow/deny decisions, and a sane llms.txt draft.

    What it does

    1. Audit crawler access to priority URLs (fetch/robots perspective).
    2. Fill a bot-policy decision matrix (what you want GPTBot, ClaudeBot, etc. to do — recommendations only).
    3. Draft llms.txt pointing at real canonical pages you control.
    4. Validate checklist before you (or your host) publish.

    What's in the zip

    • SKILL.md — crawler audit + llms.txt pipeline
    • Crawler access audit template
    • llms.txt draft template
    • Bot policy decision matrix
    • robots fetch + llms.txt validate checklists
    • EXAMPLE walkthrough
    • README + cover art

    What you bring

    Domain, list of key URLs, and access to edit robots.txt / root files (or a ticket to whoever can). The skill drafts; it does not SSH into your server.

    For / not for

    For: SEOs and founders aligning technical access with AI-visibility goals.

    Not for: Bypassing paywalls, cloaking, or ranking guarantees.

    Companion skills on Agensi (HubDev)

    Pairs with Citation Hub, Schema Markup, and Brand Citation Audit. Included in the HubDev GEO/AEO Skill Stack.

    Sample output (real public sites, abridged)

    Sites: stripe.com and anthropic.com (first audited 2026-11-06, re-checked 2026-11-13; neither is a HubDev client; files may change after these dates)

    Checkstripe.comanthropic.com
    robots.txt200, fetched200, fetched
    Named AI crawler rules (GPTBot, ClaudeBot, PerplexityBot, Google-Extended)None; falls back to the User-agent: * groupNone; "User-Agent: * / Allow: /"
    llms.txtPresent (200); starts with a one-line summary, then sections such as PaymentsNot found (404)

    What the skill concludes: neither site blocks these AI crawlers by name, so access depends on the wildcard rules. Stripe already publishes an llms.txt, so the job is a review. Anthropic has none, so the skill would draft one from the site's main pages for the owner to approve. Every draft is clearly labeled as an example until the owner publishes it.

    The audit only reports what the files say. It does not promise crawling, indexing, or citations.

    Cannot bypass paywalls or cloaking. Requires manual verification of page existence to avoid 404s. llms.txt is emerging guidance and not a ranking guarantee.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 10 days ago

    Permissions

    No special permissions declared or detected

    Listed10 days ago
    Updated3 days ago

    What's inside

    Frequently Asked Questions

    Treat it as emerging AI guidance for crawlers/agents, not a ranking guarantee.

    No. It drafts recommendations; you or your host publish the files.

    Basic comfort editing robots.txt / llms.txt helps; the skill writes the drafts.

    New in LLM Ops