Summary
Straightforward Python script that fetches web pages and converts them to markdown. Good for when you need to read articles or extract content from URLs, especially if Claude's built-in fetching hits network restrictions. It strips out navigation and ads, preserves formatting like headings and code blocks, and can handle multiple URLs in parallel with basic shell scripting. Supports custom timeouts and has a raw HTML mode if you need it. Won't help with JavaScript-heavy sites since it's just HTTP requests and HTML parsing, but for static content it gets the job done without dependencies on external services.
Install to Claude Code
npx -y skills add zephyrwang6/myskill --skill web-scraper --agent claude-codeInstalls into .claude/skills of the current project.
Fastest browser for AI agents to run web automation tasks, always free.
Download Free life-time →
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →
Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →
Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →
Notes, actions and memory. Without a meeting bot. First month 100% off.
Download for free →
Files
SKILL.mdView on GitHub
Web Scraper
Fetch web page content and convert to clean markdown format.
Usage
Run the fetch script to get web content:
python3 scripts/fetch_url.py <url> [options]
Options
--timeout <seconds>: Request timeout (default: 30)--max-length <chars>: Maximum output length (default: 100000)--raw: Output raw HTML instead of markdown
Examples
Fetch single URL:
python3 scripts/fetch_url.py "https://example.com/article"
Fetch with custom timeout:
python3 scripts/fetch_url.py "https://example.com/article" --timeout 60
Fetch multiple URLs in parallel:
for url in "https://url1.com" "https://url2.com"; do
python3 scripts/fetch_url.py "$url" &
done
wait
Workflow
- Single URL: Run
fetch_url.pywith the URL - Multiple URLs: Run multiple fetch commands in parallel using background processes
- Handle errors: If a URL fails, check:
- Network connectivity
- URL validity
- Website may block automated requests (try different User-Agent or use browser automation)
Output Format
The script converts HTML to clean markdown:
- Headings →
#,##,###, etc. - Lists →
-for unordered,1.for ordered - Bold/Italic →
**bold**,*italic* - Code blocks preserved
- Navigation, footer, and ads removed
Troubleshooting
403 Forbidden: Website blocks automated requests. Consider:
- Some sites require JavaScript rendering (not supported by this script)
- Try accessing from a different network
Timeout errors: Increase timeout with --timeout 60
Empty content: Website may require JavaScript to render content
Featured
Fastest browser for AI agents to run web automation tasks, always free.
Download Free life-time →belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →Free an open-source skills that make AI coding agents easier to understand, verify, and control.
Download for free →Connect your Claude agent to live crypto prices and trading routes via 1inch
Get the MCP →Notes, actions and memory. Without a meeting bot. First month 100% off.
Download for free →First SeenJun 3, 2026
