- Home
- Skills
- APIs & Backend
- Web Scraper: Polite, Reliable Data Extraction
Works with the AI Agents you already use
Web Scraper: Polite, Reliable Data Extraction
Web Scraper makes your agent extract data like a careful data engineer.
Free
Web Scraper: Polite, Reliable Data Extraction
Example session with this skill installed
Get all products (name, price, availability, URL) from this supplier catalog into a spreadsheet: https://example-supplier.com/catalog
- Read your context and instructions
- Compiled the web scraper
Checks
- No public API;
sitemap.xmllists 412 product URLs, but category pages already contain all fields -> scrape 18 category pages instead of 412 product pages (23x fewer requests) robots.txt: catalog allowed,/cartand/accountdisallowed,Crawl-delay: 2-> delay set to 2 s- Content is in the initial HTML (no JS needed); each card also has JSON-LD
Productwith price and availability -> used it as the primary source (more stable than CSS classes)
Selector map
| Field | Source |
|---|---|
| name | JSON-LD name (fallback .card h3) |
| price | JSON-LD offers.price + priceCurrency |
| availability | JSON-LD offers.availability -> in_stock / out_of_stock |
| url | .card a@href (absolute) |
| pagination | a[rel=next] |
Run
python scripts/polite_scrape.py --urls categories.txt --item ".card" \
--field name="h3" --field price=".price" --field url="a@href" \
--next "a[rel=next]@href" --delay 2 --out raw.csv
Then merged with the JSON-LD extraction and cleaned.
Result: products.xlsx
- 396 products from 18 pages in 41 s (cached; re-runs are instant)
- 16 duplicates removed (products listed in two categories)
- Prices
1.234,50 €->1234.50+currency=EUR; availability normalised - Empty fields: name 0 %, price 1.5 % (6 "price on request" items, flagged), url 0 %
- Spot-checked 3 random products against the live site: all match
Notes
- If the site redesigns, JSON-LD is the most likely to survive; the CSS fallback for name may break.
- For weekly price tracking: run with a fresh cache every Monday and append
scraped_at- want me to set that up?
Connects securely to your tools. The creator never sees your data.
About this skill
Get the data without hammering the site (or getting garbage)
Quick scraping scripts break on page 2, mangle accents and prices, ignore robots.txt and get blocked. Web Scraper makes your agent extract data like a careful data engineer.
What it does
- Official sources first: public APIs, RSS, sitemaps, exports, open data, and data already embedded in the page (JSON-LD,
__NEXT_DATA__,__NUXT__, XHR endpoints). - Legitimacy check: robots.txt and terms; no login/paywall/CAPTCHA bypass, no block evasion, no bulk personal data - says so clearly when a request crosses the line.
- Inspect before coding: record container, field selectors (stable hooks over generated classes), pagination pattern, static vs JS-rendered.
- Right tool: included
polite_scrape.py(requests + BeautifulSoup) for static pages, Playwright for JS pages, direct JSON APIs,pandas.read_htmlfor tables, Scrapy for large crawls. - Polite & robust: real User-Agent, robots.txt + Crawl-delay, 1 request at a time with delay, backoff on 429/5xx with Retry-After, page cache, retries, resume.
- Clean & validated output: locale-aware numbers, ISO dates, absolute URLs, dedupe,
_sourceURL, empty-field report and spot checks against the live page. - Re-runnable: script + config, schedule notes and which selectors are fragile.
For price tracking, catalogs, listings, directories, research datasets and HTML tables to spreadsheets.
Includes
- SKILL.md workflow
- scripts/polite_scrape.py - tested scraper: robots.txt, rate limit, retries, cache, pagination, CSV/JSON, encoding fix
- references/selectors.md - hidden structured data, selector tips, pagination patterns, Playwright snippet, cleaning helpers
Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI and any SKILL.md-compatible agent. Requires Python 3.
Does not bypass CAPTCHAs, paywalls, or anti-bot shields. No support for scraping behind logins or collecting private personal data. Requires Python environment for script execution.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
This skill has been used 12 times.
Trust & safety
Security scan
Security scan passed 9 days ago
- Free with an Agensi account