Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    Stealth Web Scraper

    1

    Scrape pages that block plain requests: impersonate real browsers with curl_cffi, keep a session cookie jar, rotate to Playwright when JavaScript rendering or a

    Free

    4 usesSecurity scanned
    Stealth Web Scraper

    Stealth Web Scraper

    Example session with this skill installed

    Scrape the product names and prices from https://example-store.com/deals which uses a heavy anti-bot wall and loads content via JavaScript.

    • Read your context and instructions
    • Compiled the stealth web scraper

    Starting Stage 3 escalation with Playwright. Waiting for '.product-grid' selector...

    Data extracted

    1. "Pro Wireless Mouse" | $49.99
    2. "Mechanical Keyboard" | $89.00
    3. "USB-C Hub" | $25.50
      Crawl complete. 3 items saved to page.html.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Standard scraping requests often trigger 403 Forbidden errors, CAPTCHA walls, or return empty HTML because the target site detects non-browser TLS fingerprints or requires JavaScript execution.

    What it does

    • Escalates through a three-stage "scraping ladder" from simple headers to full browser emulation to minimize overhead.
    • Impersonates real Chrome TLS fingerprints and header ordering to bypass sophisticated bot detection systems.
    • Maintains persistent session cookie jars across requests to handle multi-step navigation or authentication.
    • Renders JavaScript-heavy pages and waits for specific DOM elements using headless Chromium.
    • Extracts structured data from raw HTML using CSS selectors and XPath.

    Frameworks & tools

    Python, curl_cffi, Playwright, Chromium, and Parsel.

    Why this beats prompting it yourself

    Writing reliable scraping logic requires deep knowledge of TLS fingerprinting and browser emulation. This skill implements a proven escalation strategy that avoids over-using heavy resources like Playwright when lightweight impersonation is sufficient.

    Use cases

    • Extracting product listings from e-commerce sites with anti-bot protection.
    • Gathering data from single-page applications (SPAs) that require script execution.
    • Collecting research data from sites that block standard Python Requests or cURL.
    • Scraping paginated content that requires session and cookie persistence.

    Known limitations

    Not designed for bypassing logins, paywalls via credential abuse, or high-volume abusive scraping. Requires Python environment with Playwright binaries installed.

    Requires local installation of curl_cffi and Playwright. Not intended for bypassing paywalls or credential-based authentication.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    4 uses

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 3 months ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Browser
    Network Access
    Listed3 months ago

    What's inside

    Frequently Asked Questions

    This skill uses a 'staged escalation' strategy, starting with lightweight TLS impersonation (curl_cffi) and only moving to full browser rendering (Playwright) if necessary to bypass detection or execute JavaScript.

    The package includes the core scraping logic, configuration templates for various detection levels, and documentation on how to integrate the 'scraping ladder' into your existing Python workflows.

    This skill is designed for Python-based AI agents and requires a Python 3.8+ environment with the Playwright binaries installed; it is compatible with most custom-built LLM agents and automation frameworks.

    Yes, the skill is designed to manage persistent session cookie jars, allowing your agent to navigate through multi-step processes or maintain a 'logged-in' state across different pages.

    The skill is built to be modular; you can easily update the underlying Playwright and curl_cffi libraries as new browser fingerprints are released to maintain a high bypass success rate.

    New in Testing & Debugging