- Home
- Skills
- Workflow Automation
- Scrape Protected Websites
Works with the AI Agents you already use
Scrape Protected Websites
Fetch pages that block normal requests (403, captcha, JavaScript-only) as HTML, clean Markdown or AI-parsed JSON.
Free
Scrape Protected Websites
Example session with this skill installed
Fetch the product price and availability for https://example.com/item-123 as structured JSON using the US proxy.
- Read your context and instructions
- Compiled the scrape protected websites
I have successfully fetched the structured data for the requested URL using a US-based proxy. The ScrapeUnblocker API bypassed the site's protection and extracted the following fields.
Extracted Product Data
| Field | Value |
|---|---|
| URL | https://example.com/item-123 |
| Page Type | product_listing |
| Title | Wireless Noise Cancelling Headphones - Model X |
| Price | $299.99 |
| Currency | USD |
| Availability | In Stock |
| Rating | 4.8/5 |
| Review Count | 1,240 |
| Source | scrape-unblocker-ai |
| Data Extracted | true |
Next steps
- Map these JSON fields to your internal database schema.
- If you need the full product description for an LLM, run the command again with
markdown=true. - Check the
X-Origin-Statusheader if you encounter 404 errors on specific SKU variations.
Connects securely to your tools. The creator never sees your data.
About this skill
Standard HTTP clients fail when faced with Cloudflare, 403 Forbidden errors, or JavaScript-heavy SPAs. This skill routes requests through a headless browser environment to bypass bot detection and retrieve clean data.
What it does
- Bypasses bot protection by using real browser rendering to circumvent 403, 429, and captcha blocks.
- Extracts structured JSON using AI-parsed fields for products, articles, and listings without manual selectors.
- Converts to Markdown to provide LLM-friendly content, stripping boilerplate for RAG or summarization.
- Handles Geo-restrictions via residential proxies, allowing access to localized content using ISO country codes.
- Waits for dynamic content by observing CSS selectors or XPaths before returning the page source.
How it works
- Configure access by setting the
SCRAPEUNBLOCKER_KEYenvironment variable with your API credentials. - Execute requests via the bundled Python helper script or direct curl commands to the API endpoint.
- Specify output format by setting parameters for raw HTML, Markdown, or parsed JSON data.
- Handle responses by checking for the
X-Origin-Statusheader and verifying content extraction status.
Frameworks & tools
The skill utilizes the ScrapeUnblocker API and a Python helper script. It supports integration via official scrapeunblocker SDKs for Python and Node.js, requiring only the standard library for the base implementation.
Why this beats prompting it yourself
Most LLMs cannot browse the live web effectively when faced with sophisticated anti-bot headers or JavaScript requirements. This skill provides the specialized infrastructure needed to fetch the raw data that prompts alone cannot reach.
Use cases
- Competitor monitoring to track prices and availability on protected retail sites.
- RAG pipeline ingestion to fetch clean, distraction-free Markdown from news articles.
- Market research to gather structured data from localized search results across different countries.
- Data archival to capture fully rendered HTML from sites that rely heavily on React or Vue.
Known limitations
Requires a ScrapeUnblocker API key. Does not support bypass of logins or paywalls. Parameters like markdown and parsed_data cannot be combined in a single request.
Requires ScrapeUnblocker API key. No login/paywall bypass. Specific formats (Markdown/JSON) cannot be combined in one call. Max timeout is 60s.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free forever
No account required to browse
Trust & safety
Security scan
Security scan passed 13 days ago
- Free with an Agensi account
Requires
Permissions
Allowed Hosts