
web-and-retrieval
Web content fetching, browser automation, and URL handling tools.
- version
- 1.0.4
- GitHub stars
- 404
Install
/plugin install web-and-retrieval@oaustegard-claude-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add oaustegard/claude-skills.
Contents
Discovered from sourceSkills 3
Fetching Blocked URLs
Retrieve readable content from URLs that web_fetch cannot access. Jina AI's reader service renders JavaScript, bypasses soft blocks, and returns clean markdown.
Activation Triggers
Invoke this skill immediately when web_fetch returns:
- 403 Forbidden or access denied
- Paywall or login wall indicators
- Empty, garbled, or truncated content
- JavaScript-heavy SPA failures
- Timeout errors
Core Command
curl -s --max-time 30 "https://r.jina.ai/TARGET_URL"
The service returns markdown with page title, body text, and preserved links.
Retry Pattern
Jina's backend has ~10% intermittent failures. Use retry logic to achieve 99%+ success:
for attempt in 1 2 3; do
result=$(curl -s --max-time 30 "https://r.jina.ai/TARGET_URL" 2>&1)
echo "$result" | grep -q "upstream connect error" || { echo "$result"; break; }
[ $attempt -lt 3 ] && sleep 1
done
Workflow Integration
- Primary: Use web_fetch (native tool)
- Fallback: This skill with retry when web_fetch fails
- Escalate: Request user assistance only after retry exhaustion
Attempt this fallback before asking users to copy-paste content manually.
Output Format
Jina returns structured markdown:
Title:page titleURL Source:original URLMarkdown Content:extracted body text, links preserved
Limitations
- Long pages may truncate
- Sites blocking all scrapers remain inaccessible
- Login-required content limited to public portions
- Real-time dynamic content may not render
Domain Access
r.jina.ai is whitelisted in Claude container network configuration.
Delivery Instructions
When triggered:
- Copy the asset to outputs:
cp assets/hello-demo.html /mnt/user-data/outputs/
- Provide the computer link:
[View hello-demo.html](computer:///mnt/user-data/outputs/hello-demo.html)
- Add one-line description: "Interactive HTML demo with bookmarklet showing token-efficient artifact delivery."
DO NOT:
- Generate HTML from scratch
- Explain the file contents in detail
- Add verbose commentary
The asset contains a complete, styled HTML page with:
- Bookmarklet demonstration
- Usage instructions
- Token efficiency explanation
Token efficiency: This pattern saves ~500 tokens vs. generating explanatory text each time.
Using webctl in Claude.ai Containers
Browser automation using webctl CLI with automatic proxy authentication handling for Claude.ai's egress-controlled environment.
When This Skill Applies
- User requests browser automation, web scraping, or page interaction
- webctl commands fail with
ERR_TUNNEL_CONNECTION_FAILED - Playwright/Chromium needs to work through authenticated HTTP proxy
Core Problem
Claude.ai containers route traffic through an authenticated egress proxy (HTTP_PROXY env var with JWT credentials). Chromium doesn't properly handle proxy authentication for HTTPS CONNECT tunnels, causing all HTTPS navigation to fail even though curl works.
Solution
A local forwarding proxy (port 18080) intercepts Chromium connections and injects Proxy-Authorization headers before forwarding to the real egress proxy.
Setup Procedure
1. Install webctl
pip install webctl --break-system-packages
webctl setup # Downloads Chromium if needed
2. Deploy Auth Proxy Module
Copy scripts/auth_proxy.py to webctl's daemon directory:
cp /mnt/skills/user/using-webctl/scripts/auth_proxy.py \
/usr/local/lib/python3.12/dist-packages/webctl/daemon/
3. Patch Session Manager
Apply this patch to /usr/local/lib/python3.12/dist-packages/webctl/daemon/session_manager.py.
Recent webctl versions split context creation into two branches (mobile emulation vs desktop). Both must receive the proxy config. Find the block (search for cfg = WebctlConfig.load() followed by the if mode == "unattended" and cfg.mobile_emulation: branch — around line 190):
cfg = WebctlConfig.load()
if mode == "unattended" and cfg.mobile_emulation:
# Mobile emulation: cleaner pages, fewer ads, simpler layouts
device = self._playwright.devices["Pixel 7"]
context = await browser.new_context(
storage_state=storage_state,
**device,
)
else:
context = await browser.new_context(
storage_state=storage_state, viewport={"width": 1280, "height": 720}
)
Replace with:
cfg = WebctlConfig.load()
from .auth_proxy import get_local_proxy_url
proxy_url = get_local_proxy_url()
proxy_config = {"server": proxy_url} if proxy_url else None
if mode == "unattended" and cfg.mobile_emulation:
# Mobile emulation: cleaner pages, fewer ads, simpler layouts
device = self._playwright.devices["Pixel 7"]
context = await browser.new_context(
storage_state=storage_state,
proxy=proxy_config,
**device,
)
else:
context = await browser.new_context(
storage_state=storage_state, viewport={"width": 1280, "height": 720},
proxy=proxy_config
)
If the file only has a single new_context call (older webctl), inject proxy=proxy_config into that one and place the three proxy_url lines just above it.
4. Verify
webctl start --mode unattended
webctl --quiet navigate "https://github.com"
webctl snapshot --interactive-only --limit 10
webctl stop
Quick Reference
Session Management
webctl start --mode unattended # Headless browser
webctl stop # Full shutdown (use --keep-daemon to leave daemon running)
webctl status # Current state + console error counts
Navigation
webctl navigate "https://..."
webctl back / webctl forward / webctl reload
Observation
webctl snapshot --interactive-only --limit 30 # Buttons, links, inputs
webctl snapshot --within "role=main" # Scope to container
webctl query "role=button name~=Submit" # Debug queries
webctl screenshot --path shot.png
Interaction
webctl click 'role=button name~="Submit"'
webctl type 'role=textbox name~="Email"' "user@example.com"
webctl type 'role=textbox name~="Search"' "query" --submit
webctl select 'role=combobox name~="Country"' --label "Germany"
Query Syntax
role=button— By ARIA rolename~="partial"— Contains (preferred, more robust)name="exact"— Exact matchnth=0— Select first when multiple matches
Wait Conditions
webctl wait network-idle
webctl wait 'exists:role=button name~="Continue"'
webctl wait 'url-contains:"/dashboard"'
Troubleshooting
ERR_TUNNEL_CONNECTION_FAILED
Auth proxy not loaded. Verify:
auth_proxy.pyexists in webctl daemon directory- Session manager is patched
- Restart daemon:
webctl stop && webctl start --mode unattended
Multiple matches error
Add specificity or use nth=0:
webctl click 'role=link name="Sign in" nth=0'
Verify proxy is running
netstat -tlnp | grep 18080
Limitations
- Patch is session-local (container resets clear it)
- Only allowed domains in network config are accessible
- No Firefox support (download blocked by egress policy)
Domain Restrictions
Check <network_configuration> in system prompt for allowed domains. Common allowed: *.github.com, *.bsky.app, allowed API endpoints.
Output Filtering
Reduce context consumption:
webctl snapshot --interactive-only --limit 30 # Cap elements
webctl snapshot | grep -i "submit" # Unix filtering
webctl --quiet navigate "..." # Suppress events
Sponsors
Details
- Marketplace
- oaustegard/claude-skills
- Source
- plugins/web-and-retrieval
- Category
- Web And Retrieval
- License
- MIT
- Keywords
- fetching-blocked-urls, hello-demo, using-webctl
More from oaustegard/claude-skills
All 14- ai-and-reasoningv2.9.1
- code-intelligencev2.6.0
- data-and-visualizationv1.0.0
- deciding-with-confidencev0.4.0
- declaudingv0.14.0
- delegating-with-contextv0.3.0
- development-toolsv1.2.1
- environment-and-configv2.0.0
- github-and-gitv2.0.1
- knowledge-and-memoryv1.3.0
- media-processingv2.3.0
- skill-developmentv2.6.0
- social-and-lifestylev1.0.0