Works with the AI Agents you already use
Pagerender Tools
Audit SEO crawlability and AI bot access for any website using CLI and MCP tools.
Free
Pagerender Tools
Example session with this skill installed
Check why https://pagerender.io might be struggling with SEO and if GPTBot is allowed to crawl it.
- Read your context and instructions
- Compiled the pagerender tools
I have analyzed the site's crawlability and its robots.txt rules for AI agents.
Crawlability Audit & AI Access Report
| Tool | URL / Domain | Status | Result |
|---|---|---|---|
analyze_url | https://pagerender.io | 200 OK | verdict: good |
check_ai_crawlers | pagerender.io | Success | GPTBot: Allowed |
Detailed Page Analysis
| Metric | Value | Meaning |
|---|---|---|
| App Shell | false | Content is visible without JavaScript. |
| Word Count | 412 | Sufficient text content for indexing. |
| Crawlable Links | 12 | Search engines can discover other pages. |
| JS Links | 0 | No broken navigation patterns detected. |
| Noindex | false | The page is permitting indexing. |
| Crawler Blocked | false | No bot protection interference detected. |
AI Crawler Access
The robots.txt file for pagerender.io was successfully parsed.
- Allowed Bots:
gptbot,claudebot,ccbot,bingbot, and 17 others. - Blocked Bots: None.
Next steps
- Run a full sitemap audit using
npx @pagerender/cli audit --sitemap https://pagerender.io/sitemap.xmlto check forpoorverdicts on inner pages. - Monitor
jsLinkscount on future updates to ensure navigation remains crawlable. - Check Google Search Console to see if the "good" verdict translates to successful indexing.
Connects securely to your tools. The creator never sees your data.
About this skill
JavaScript-heavy sites often look perfect in a browser but appear as empty shells to search engines and AI agents. This skill provides the diagnostic tools needed to identify why pages aren't indexing or why crawlers are being blocked by bot protection.
What it does
- Crawler simulation extracts exactly what a search engine or AI bot sees before JavaScript execution.
- SEO health audits scans sitemaps to identify empty app shells, low word counts, and missing meta tags.
- Robots.txt analysis checks access permissions for 21 specific AI crawlers including GPTBot and ClaudeBot.
- CI/CD integration provides exit codes and CLI flags to fail builds if a release breaks crawlability.
- Firewall detection identifies when bot protection services are accidentally blocking legitimate search crawlers.
How it works
- Analyze a URL using the
analyze_urltool to get a diagnostic verdict on a single page's crawlability. - Sample a site with
analyze_siteto check the homepage and up to 7 linked pages for common issues. - Verify AI access via
check_ai_crawlersto see which specific AI agents are allowed or blocked by robots.txt. - Run a full audit by providing a sitemap URL to the CLI to generate a health report for the entire domain.
Frameworks & tools
Built for Node 20+ environments. Works with any web framework including React, Vue, Angular, Next.js, and Nuxt.js. Integrates with MCP clients like Claude Code and Cursor.
Why this beats prompting it yourself
Standard LLMs guess crawlability based on their training data, which is often outdated or incomplete. This skill uses live CLI and MCP tools to fetch real HTML and verify current robots.txt rules, giving you factual data instead of hallucinations.
Use cases
- Debugging why a Next.js or SPA site is missing from Google Search results.
- Verifying that AI assistants like Claude or ChatGPT can actually parse your documentation.
- Preventing SEO regressions by adding crawl checks to a GitHub Actions pipeline.
- Auditing site migrations to ensure internal links remain crawlable by search bots.
Known limitations
The tools do not execute JavaScript, as they are designed to match the first-pass behavior of crawlers. It cannot audit pages behind a login or private authentication.
Does not execute JavaScript. Cannot access pages behind authentication. Sitemap audits follow indexes only one level deep. CLI audit limit defaults to 500 URLs.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 7 days ago
- Free with an Agensi account
Requires