Summary
A proper scraping cascade that tries trafilatura first, falls back to requests with rotated user agents, then escalates to Playwright with stealth mode if the site runs JavaScript or basic anti-bot checks. The code is clean and tracks which method succeeded. The anti-bot landscape section is honest about what playwright-stealth actually handles (navigator.webdriver patches, fingerprint evasion) versus what it doesn't (TLS fingerprinting, Cloudflare Turnstile). The async Playwright variant for Jupyter notebooks is a nice touch since sync Playwright breaks in notebook event loops. This won't beat DataDome or sophisticated bot management, but it covers the 80% case where you just need content extraction with reasonable resilience.
Install to Claude Code
npx -y skills add jamditis/claude-skills-journalism --skill web-scraping --agent claude-codeInstalls into .claude/skills of the current project.
Keeps your Mac awake while Claude Code, Codex or Cursor works. Lets it sleep when they are done.
Try free for 7 days →
Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →
Block distracting apps from your iPhone permanently without a 3rd party app. Free and open source.
Block now (100% free) →
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →
Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →Files
SKILL.md
Select a file.
Featured
Keeps your Mac awake while Claude Code, Codex or Cursor works. Lets it sleep when they are done.
Try free for 7 days →Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →Block distracting apps from your iPhone permanently without a 3rd party app. Free and open source.
Block now (100% free) →belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →
