Summary
A proper scraping cascade that tries trafilatura first, falls back to requests with rotated user agents, then escalates to Playwright with stealth mode if the site runs JavaScript or basic anti-bot checks. The code is clean and tracks which method succeeded. The anti-bot landscape section is honest about what playwright-stealth actually handles (navigator.webdriver patches, fingerprint evasion) versus what it doesn't (TLS fingerprinting, Cloudflare Turnstile). The async Playwright variant for Jupyter notebooks is a nice touch since sync Playwright breaks in notebook event loops. This won't beat DataDome or sophisticated bot management, but it covers the 80% case where you just need content extraction with reasonable resilience.
Install to Claude Code
npx -y skills add jamditis/claude-skills-journalism --skill web-scraping --agent claude-codeInstalls into .claude/skills of the current project.
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →
Plug Mailtrap into your AI workflow and let it handle the email.
Connect Mailtrap MCP →
Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →
Make coding agent sessions - Searchable, Shareable, Vendor-neutral & Scored.
Try For Free →
Your agent targets a perfect 10 Code Health score. Deterministic. Every commit.
Try For Free →
Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →Files
SKILL.md
Select a file.
Featured
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →Plug Mailtrap into your AI workflow and let it handle the email.
Connect Mailtrap MCP →Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →Make coding agent sessions - Searchable, Shareable, Vendor-neutral & Scored.
Try For Free →Your agent targets a perfect 10 Code Health score. Deterministic. Every commit.
Try For Free →Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →
