Works with the AI Agents you already use
onlinewebdirectory
Predict bot blocks, identify required fetch strategies, and discover machine-readable interfaces via a shared registry.
Free
onlinewebdirectory
Example session with this skill installed
I need to pull the product categories from g2.com. Before you fetch anything, check how g2.com treats agents and pick the right way in.
- Read your context and instructions
- Compiled the onlinewebdirectory
g2.com: checked before fetching
Short answer: don't scrape the pages. g2.com puts a Cloudflare bot challenge in front of plain HTTP clients, but it publishes an llms.txt, so I read that instead.
What the directory says about g2.com
| Fact | Value | Source |
|---|---|---|
| Access | Bot challenged | Registry check, 25 Oct 2026 |
| Bot wall | Cloudflare | Registry check |
| Robots rules | Restricts some paths | Registry check |
| Machine-readable alternative | llms.txt at /llms.txt | Registry check |
Fetch strategy returned: use_machine_interface, prefer /llms.txt, expect a Cloudflare challenge on normal pages.
What I did
- Skipped the HTML pages, which would have hit the Cloudflare challenge and wasted a headless browser run.
- Fetched
https://g2.com/llms.txtwith a plain HTTP request. - Extracted the category sections from it.
After fetching
I reported what I saw back to the directory (llms.txt present and readable, signed with my key), so the next agent that checks g2.com gets a better-verified answer.
Next steps
- If you need detail that llms.txt doesn't cover, I can use a real browser for those specific pages only.
- Want me to check the other sites on your list the same way first?
Connects securely to your tools. The creator never sees your data.
About this skill
Scraping the web with agents often leads to wasted resources, blocked requests, and unnecessary DOM parsing. Websites change their bot policies without notice, leaving your scrapers to fail silently or hit expensive paywalls.
This skill provides a programmatic interface to a shared registry of how websites treat agents. It allows your agents to check a site's access posture before attempting a fetch, ensuring they choose the right client—whether that is a plain HTTP request, a headless browser, or a machine-readable interface like llms.txt. By contributing observations back to the registry, your agents help maintain a crowd-verified map of the web's bot-friendliness.
What it does
- Pre-fetch lookup checks if a domain blocks agents or requires JavaScript before you spend compute.
- Client selection identifies the specific bot wall (Cloudflare, DataDome, etc.) to help choose the correct impersonation strategy.
- Interface discovery locates
llms.txt, RSS feeds, or sitemaps so you can skip HTML scraping entirely. - Crowd verification tracks whether observations are verified by multiple agents or are single-report hints.
- Observation reporting lets agents sign and submit facts about rate limits, challenges, and robots.txt posture.
How it works
- Lookup origin: Call
lookupfor one or more domains to retrieve the currentstrategy.fetchandaccess_posture. - Evaluate facts: Check
strategy.basisto determine if the data is crowd-verified or a pending registry check. - Execute fetch: Use the recommended client (e.g.,
headless_chromeorplain_http) based on the lookup results. - Submit report: Record what actually happened during the fetch to update the registry and earn points.
Frameworks & tools
Built with Node.js. It supports integration via a CLI wrapper (owd.mjs), a remote MCP server, or a direct REST API.
Why this beats prompting it yourself
LLMs have outdated knowledge of website structures and bot defenses. This skill provides real-time, crowd-sourced intelligence that prevents agents from repeatedly hitting known blocks or missing hidden machine-readable endpoints.
Use cases
- Cost optimization by skipping domains known to have hard paywalls or login requirements.
- Reliable scraping by identifying exactly which sites require a headless browser for JavaScript execution.
- Data enrichment by automatically finding and utilizing
llms.txtor sitemaps instead of noisy HTML.
Known limitations
Requires Node.js for the CLI version. Facts can go stale after 14 to 90 days, requiring fresh reports to maintain accuracy. Free lookups are limited to 100 per day per key.
Requires Node.js for CLI. Facts expire after 14 to 90 days. Daily lookup quota applies. Data reliability depends on recent crowd reports.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 17 days ago
- Free with an Agensi account
Requires
Permissions
Allowed Hosts
File Scopes