Blocked Page Recovery
Recover content from 403/429-blocked, paywalled and Cloudflare-walled pages via third-party archives and API endpoints instead of retry-looping.
Free
Works with the AI Agents you already use
About this skill
Blocked-Page Recovery
When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall, or a bot-detection interstitial — don't give up and don't loop on the same URL. Third-party services often hold a copy of the page. Work down this ladder, cheapest first.
The ladder
1. Wayback Machine — archive.org "available" API (snapshot + timestamp)
2. archive.today — domain rotation: archive.ph -> .md -> .li -> .is
3. Jina Reader — only if JINA_API_KEY is set (live server-side render)
4. API-first pivot — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser — browser automation as the last, most expensive resort
Run it in one shot with the bundled script:
python3 scripts/recover_page.py "https://example.com/blocked-article" --json
The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance.
Provenance discipline (non-negotiable)
Every recovered copy carries a provenance you MUST preserve when citing:
| Route | Provenance | How to cite |
|---|---|---|
| Wayback / archive.today | snapshot | Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale. |
| Jina Reader | live | Server-side re-render of the live page; cite normally. |
| Live fetch / browser | live | Cite normally. |
If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer — say so explicitly and note its age.
Manual routes
1. Wayback Machine (best provenance, try first)
# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url
For enumerating many snapshots (or recovering deleted pages), the CDX index:
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"
CDX intermittently returns 503 under load — if it does, fall back to the
available API; don't retry-hammer it.
Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).
2. archive.today (paywalls, deleted content)
User-submitted archives — often has paywalled news articles Wayback lacks. Rate-limits aggressively (429) and rotates domains, so iterate:
for d in archive.ph archive.md archive.li archive.is; do
curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
-w "%{http_code}" && break
done
Validate the body, not the status code — a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.
3. Jina Reader (requires JINA_API_KEY)
r.jina.ai re-renders the live page in a real browser server-side and
returns markdown. Anonymous access is dead (401 -> Turnstile); a key is
required:
curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"
Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.
4. API-first pivot
WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:
/api/...,/graphql, or.jsonvariants of the page URL- An RSS/Atom feed (
/feed,/rss,<link rel="alternate">in any copy you did recover) - A sitemap (
/sitemap.xml) revealing canonical URLs that may not be gated
Fake successes — routes that LIE
These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:
- Google Cache is dead (since mid-2024).
webcache.googleusercontent.comreturns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it. - AMP caches (
*.cdn.ampproject.org) mostly return a ~300-byte<title>Redirecting</title>meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop. - Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.
Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").
Proxy relays: don't
Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on — provenance is unverifiable. Prefer archives, which at least timestamp their copies.
Requirements
Python 3.8+, requests (falls back to urllib). No API keys required
(Jina route is skipped automatically without JINA_API_KEY).
Cannot bypass strongly authenticated pages or non-archived content. Success depends on third-party archives having the page.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
This skill has been used 4 times.
Trust & safety
Security scan
Security scan passed 1 month ago
- Free with an Agensi account
Requires
Permissions
Allowed Hosts
Skill contacts only these hosts and sends only the target URL. No third-party re-render service (r.jina.ai removed in v1.1.0).