Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Pagerender Tools

    1

    Audit SEO crawlability and AI bot access for any website using CLI and MCP tools.

    Free

    Security scanned
    Pagerender Tools

    Pagerender Tools

    Example session with this skill installed

    Check why https://pagerender.io might be struggling with SEO and if GPTBot is allowed to crawl it.

    • Read your context and instructions
    • Compiled the pagerender tools

    I have analyzed the site's crawlability and its robots.txt rules for AI agents.

    Crawlability Audit & AI Access Report

    ToolURL / DomainStatusResult
    analyze_urlhttps://pagerender.io200 OKverdict: good
    check_ai_crawlerspagerender.ioSuccessGPTBot: Allowed

    Detailed Page Analysis

    MetricValueMeaning
    App ShellfalseContent is visible without JavaScript.
    Word Count412Sufficient text content for indexing.
    Crawlable Links12Search engines can discover other pages.
    JS Links0No broken navigation patterns detected.
    NoindexfalseThe page is permitting indexing.
    Crawler BlockedfalseNo bot protection interference detected.

    AI Crawler Access

    The robots.txt file for pagerender.io was successfully parsed.

    • Allowed Bots: gptbot, claudebot, ccbot, bingbot, and 17 others.
    • Blocked Bots: None.

    Next steps

    1. Run a full sitemap audit using npx @pagerender/cli audit --sitemap https://pagerender.io/sitemap.xml to check for poor verdicts on inner pages.
    2. Monitor jsLinks count on future updates to ensure navigation remains crawlable.
    3. Check Google Search Console to see if the "good" verdict translates to successful indexing.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    JavaScript-heavy sites often look perfect in a browser but appear as empty shells to search engines and AI agents. This skill provides the diagnostic tools needed to identify why pages aren't indexing or why crawlers are being blocked by bot protection.

    What it does

    • Crawler simulation extracts exactly what a search engine or AI bot sees before JavaScript execution.
    • SEO health audits scans sitemaps to identify empty app shells, low word counts, and missing meta tags.
    • Robots.txt analysis checks access permissions for 21 specific AI crawlers including GPTBot and ClaudeBot.
    • CI/CD integration provides exit codes and CLI flags to fail builds if a release breaks crawlability.
    • Firewall detection identifies when bot protection services are accidentally blocking legitimate search crawlers.

    How it works

    1. Analyze a URL using the analyze_url tool to get a diagnostic verdict on a single page's crawlability.
    2. Sample a site with analyze_site to check the homepage and up to 7 linked pages for common issues.
    3. Verify AI access via check_ai_crawlers to see which specific AI agents are allowed or blocked by robots.txt.
    4. Run a full audit by providing a sitemap URL to the CLI to generate a health report for the entire domain.

    Frameworks & tools

    Built for Node 20+ environments. Works with any web framework including React, Vue, Angular, Next.js, and Nuxt.js. Integrates with MCP clients like Claude Code and Cursor.

    Why this beats prompting it yourself

    Standard LLMs guess crawlability based on their training data, which is often outdated or incomplete. This skill uses live CLI and MCP tools to fetch real HTML and verify current robots.txt rules, giving you factual data instead of hallucinations.

    Use cases

    • Debugging why a Next.js or SPA site is missing from Google Search results.
    • Verifying that AI assistants like Claude or ChatGPT can actually parse your documentation.
    • Preventing SEO regressions by adding crawl checks to a GitHub Actions pipeline.
    • Auditing site migrations to ensure internal links remain crawlable by search bots.

    Known limitations

    The tools do not execute JavaScript, as they are designed to match the first-pass behavior of crawlers. It cannot audit pages behind a login or private authentication.

    Does not execute JavaScript. Cannot access pages behind authentication. Sitemap audits follow indexes only one level deep. CLI audit limit defaults to 500 URLs.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 7 days ago

    • Free with an Agensi account

    Requires

    @pagerender/cli
    Node.js
    from
    @pagerender/mcp

    Permissions

    Terminal / Shell
    Network Access
    Listed7 days ago

    What's inside

    Frequently Asked Questions

    This skill identifies "empty shell" issues where JavaScript-heavy frameworks (like React or Next.js) prevent search engines and AI bots from seeing your content, ensuring your site is actually indexable rather than just a blank page.

    It specifically checks access for 21 different AI agents, including GPTBot, ClaudeBot, and OAI-SearchBot, ensuring your documentation and content are accessible to the latest LLM-driven search tools.

    The tools are designed for Node 20+ environments and integrate seamlessly with MCP-compatible clients like Claude Code and Cursor, as well as any standard CLI for CI/CD pipelines.

    Yes, the CLI provides specific exit codes and flags designed to fail builds in environments like GitHub Actions if a new release breaks SEO crawlability or blocks essential bots.

    To accurately simulate the first-pass behavior of search engines and AI crawlers, these tools analyze the raw HTML response and do not execute JavaScript.

    You receive the full set of MCP tools and CLI utilities, including the URL analyzer, site sampler, AI crawler verifier, and the sitemap auditing engine.

    New in LLM Ops