
doc-fetcher
Fetches, caches, and versions documentation from web sources to provide accurate, version-specific context for AI coding agents
- version
- 2.9.1
Install
/plugin install doc-fetcher@squirrelsoft-dev-toolsRun it inside Claude Code after adding the marketplace with /plugin marketplace add squirrelsoft-dev/squirrelsoft-marketplace.
Contents
Discovered from sourceSkills 5
Dependency Detector
I automatically detect your project's dependencies and suggest fetching relevant documentation to provide accurate, version-specific AI assistance.
What I Do
I analyze your project to understand what libraries you're using:
- Detect Package Managers: Identify npm, yarn, pnpm, bun
- Parse Dependencies: Read package.json, requirements.txt, go.mod, etc.
- Extract Versions: Get exact versions you're using
- Compare with Cache: Check what documentation is already cached
- Suggest Fetches: Recommend fetching missing or outdated docs
- Smart Prioritization: Focus on major frameworks first
Supported Project Types
JavaScript/TypeScript (Node.js)
Detection files:
package.json- npm/yarn/pnpm dependenciespackage-lock.json- Locked versionsyarn.lock- Yarn locked versionspnpm-lock.yaml- pnpm locked versionsbun.lockb- Bun locked versions
Parsed fields:
{
"dependencies": { ... }, // Production deps
"devDependencies": { ... }, // Dev deps (lower priority)
"peerDependencies": { ... } // Peer deps (important for plugins)
}
Prioritized libraries:
- Frameworks: next, react, vue, svelte, angular, nuxt
- Meta-frameworks: astro, remix, solid-start
- Backend: express, fastify, nest, koa
- Databases: prisma, typeorm, mongoose, drizzle
- Supabase: @supabase/supabase-js, @supabase/auth-helpers
- UI: tailwindcss, shadcn-ui, mui, chakra-ui
- State: redux, zustand, jotai, mobx
Python
Detection files:
requirements.txt- pip dependenciespyproject.toml- Poetry/modern PythonPipfile- Pipenvsetup.py- Package setupenvironment.yml- Conda
Prioritized libraries:
- Frameworks: django, flask, fastapi, tornado
- Data: pandas, numpy, scipy, matplotlib
- ML: tensorflow, pytorch, scikit-learn
- Async: asyncio, aiohttp, celery
Go
Detection files:
go.mod- Go modulesgo.sum- Checksums
Prioritized libraries:
- Frameworks: gin, echo, fiber, chi
- gRPC: grpc-go
- Databases: gorm, sqlx
Rust
Detection files:
Cargo.toml- Rust dependenciesCargo.lock- Locked versions
Prioritized libraries:
- Frameworks: actix-web, rocket, axum
- Async: tokio, async-std
- Serde: serde, serde_json
Detection Process
Phase 1: Project Scan
Scanning project for dependencies...
✓ Found package.json
✓ Found package-lock.json (exact versions available)
✓ Detected package manager: npm
Phase 2: Parse Dependencies
// Example package.json
{
"dependencies": {
"next": "^15.0.3",
"react": "^18.2.0",
"@supabase/supabase-js": "^2.39.0",
"tailwindcss": "^3.4.0"
},
"devDependencies": {
"typescript": "^5.3.0",
"@types/react": "^18.2.0"
}
}
Extracted:
Production dependencies: 4
Development dependencies: 2
Frameworks detected:
- Next.js (v15.0.3)
- React (v18.2.0)
Libraries detected:
- Supabase (v2.39.0)
- Tailwind CSS (v3.4.0)
Phase 3: Version Resolution
Resolving exact versions...
next: ^15.0.3 → 15.0.3 (from package-lock.json)
react: ^18.2.0 → 18.2.0 (from package-lock.json)
@supabase/supabase-js: ^2.39.0 → 2.39.0 (from package-lock.json)
tailwindcss: ^3.4.0 → 3.4.1 (from package-lock.json)
Phase 4: Cache Comparison
Comparing with cached documentation...
✓ next@15.0.3
Cached: v15.0.3 ✓ (matches exactly)
Skill: nextjs-15-expert (active)
✗ react@18.2.0
Cached: Not cached
Suggestion: /fetch-docs react 18.2.0
⚠ @supabase/supabase-js@2.39.0
Cached: v2.38.0 (outdated by 1 minor version)
Suggestion: /update-docs supabase
✓ tailwindcss@3.4.1
Cached: v3.4.0 (close enough, minor version)
Skill: tailwind-expert (active)
Phase 5: Recommendations
Documentation Recommendations:
Priority 1 (Critical frameworks):
1. /fetch-docs react 18.2.0
Reason: Core framework, not cached
Priority 2 (Outdated versions):
2. /update-docs supabase
Reason: Cached version (2.38.0) is outdated
Priority 3 (Optional):
3. /fetch-docs typescript 5.3.0
Reason: Dev dependency, could improve type hints
Quick setup:
Run all: /fetch-docs react && /update-docs supabase
Or automatically:
/fetch-docs --project
Smart Prioritization
I prioritize documentation fetching based on:
Priority 1: Critical Frameworks
These are essential and should be fetched immediately:
- Next.js, React, Vue, Svelte, Angular
- Django, Flask, FastAPI
- Express, Fastify, NestJS
Reasoning: These are the foundation of your application. Accurate docs are critical.
Priority 2: Major Libraries
Important but not critical:
- Supabase, Firebase, AWS SDK
- Prisma, TypeORM, Drizzle
- Tailwind CSS, MUI, Chakra UI
Reasoning: Used extensively, but usually have stable APIs.
Priority 3: Utility Libraries
Nice to have:
- Lodash, Ramda
- Day.js, Moment
- Axios, node-fetch
Reasoning: Smaller scope, less likely to have breaking changes.
Priority 4: Dev Dependencies
Lowest priority:
- TypeScript (mostly stable)
- Testing libraries
- Linters and formatters
Reasoning: Less likely to need frequent reference.
Library Mapping
Some libraries have different documentation names:
const libraryMappings = {
// npm package → documentation name
"@supabase/supabase-js": "supabase",
"@supabase/auth-helpers-nextjs": "supabase-auth",
"next": "nextjs",
"tailwindcss": "tailwind",
"@tanstack/react-query": "react-query",
"@mui/material": "mui",
"@chakra-ui/react": "chakra-ui"
};
I automatically map these when suggesting fetches.
Version Matching Strategy
How I determine if cached docs are "good enough":
const versionMatch = (cached, required) => {
const [cMajor, cMinor, cPatch] = cached.split('.');
const [rMajor, rMinor, rPatch] = required.split('.');
// Major version MUST match
if (cMajor !== rMajor) return 'mismatch';
// Minor version ideally matches
if (cMinor === rMinor) return 'exact';
if (Math.abs(cMinor - rMinor) === 1) return 'close';
return 'outdated';
};
// Results:
// 'exact' → ✓ Perfect, use cached docs
// 'close' → ⚠ Close enough, but update recommended
// 'outdated' → ⚠ Should update
// 'mismatch' → ✗ Must fetch correct major version
Auto-Activation
I automatically activate in these scenarios:
1. Project Opens
When you open a project with Claude Code:
Welcome to your Next.js project!
Detected dependencies:
- Next.js 15.0.3 ✓ (docs cached)
- React 18.2.0 ✗ (not cached)
Would you like me to fetch React documentation?
→ /fetch-docs react 18.2.0
2. New Dependency Added
When you modify package.json:
Detected new dependency: @tanstack/react-query@5.17.0
This library is not cached yet. Fetch documentation?
→ /fetch-docs react-query 5.17.0
3. /list-docs --project
When you explicitly request project analysis:
/list-docs --project
Project Dependencies Analysis:
From package.json:
✓ next@15.0.3 (cached, current)
✗ react@18.2.0 (not cached)
⚠ @supabase/supabase-js@2.39.0 (cached 2.38.0, outdated)
Recommendations:
/fetch-docs react 18.2.0
/update-docs supabase
Integration with Other Skills
I work with other doc-fetcher skills:
User opens project
↓
dependency-detector activates
↓
Detects: next@15.0.3, react@18.2.0
↓
Checks cache:
- next ✓ cached
- react ✗ not cached
↓
Suggests: /fetch-docs react
↓
User runs command
↓
llms-txt-finder checks for llms.txt
↓
doc-indexer crawls and caches
↓
doc-skill-generator creates react-18-expert
↓
dependency-detector confirms:
✓ All project dependencies documented
Configuration
Configure my behavior in doc-fetcher-config.json:
{
"dependency_detection": {
"auto_detect": true,
"auto_suggest": true,
"project_types": ["javascript", "typescript", "python", "go", "rust"],
"include_dev_dependencies": false,
"prioritize_frameworks": true,
"version_match_tolerance": "minor",
"suggest_on_project_open": true,
"suggest_on_dependency_add": true
}
}
Output Examples
Clean Project (All Cached)
Project dependency check:
✓ All dependencies have cached documentation
next@15.0.3 → nextjs-15-expert (active)
react@18.2.0 → react-18-expert (active)
@supabase/supabase-js@2.39.0 → supabase-expert (active)
You're all set! Claude has accurate docs for your entire stack.
Missing Docs
Project dependency check:
Missing documentation for 2 dependencies:
✗ react@18.2.0
Impact: High (core framework)
Command: /fetch-docs react 18.2.0
✗ @tanstack/react-query@5.17.0
Impact: Medium (data fetching library)
Command: /fetch-docs react-query 5.17.0
Quick fix:
/fetch-docs react 18.2.0 && /fetch-docs react-query 5.17.0
Or batch:
/fetch-docs --project
Mixed State
Project dependency check:
✓ Cached and current: 3 dependencies
next@15.0.3, typescript@5.3.0, tailwindcss@3.4.1
⚠ Cached but outdated: 1 dependency
@supabase/supabase-js@2.39.0
(cached: 2.38.0, outdated by 1 minor version)
Command: /update-docs supabase
✗ Not cached: 1 dependency
react@18.2.0
Command: /fetch-docs react 18.2.0
Quick fix:
/update-docs supabase && /fetch-docs react 18.2.0
Use Cases
1. New Project Setup
# Clone a repo
git clone https://github.com/example/app
cd app
# Open with Claude Code
# dependency-detector auto-runs
Detected: next@15.0.3, react@18.2.0, supabase@2.39.0
None cached. Fetch all?
/fetch-docs --project
# Claude now has accurate context for entire stack
2. Version Upgrade
# Upgrade Next.js
npm install next@latest
# dependency-detector notices change
Dependency version changed:
next: 15.0.3 → 15.1.0
Documentation needs update:
/update-docs nextjs
# Claude now knows about new features
3. Adding New Library
# Add React Query
npm install @tanstack/react-query
# dependency-detector notices
New dependency detected: @tanstack/react-query@5.17.0
Fetch documentation?
/fetch-docs react-query 5.17.0
# Claude can now help with React Query
Troubleshooting
"Dependency detection not working"
- Check package.json exists in root
- Verify package.json is valid JSON
- Check
auto_detectenabled in config
"Wrong version detected"
- Ensure lockfile is up to date (npm install)
- Check for conflicting version specifiers
- Manually specify version in /fetch-docs
"Too many suggestions"
- Disable dev dependencies in config
- Increase version tolerance
- Manually select which to fetch
"Library not recognized"
- Check library mapping in code
- Use custom URL: /fetch-docs mylib --url https://...
- Submit issue to add library to mappings
See Also
/list-docs --project- View project documentation status/fetch-docs --project- Fetch all missing project docs/update-docs --project- Update all project docsdoc-indexerskill - Performs the actual fetching
Documentation Fetcher Skill
You have been activated because the user wants to fetch, update, or manage library documentation.
Your Task
Use the doc-crawler subagent to fetch documentation and generate skills for the requested library.
How to Proceed
Step 1: Parse the User's Request
Extract from the user's message:
- Library name (e.g., "Next.js", "React", "Axios", "Tailwind")
- Version (optional - e.g., "15", "19", "latest")
- Custom URL (optional - if user provides a specific docs URL)
Step 2: Spawn the Doc-Crawler Agent
Use the Task tool to spawn the doc-crawler subagent with a clear prompt:
Fetch the [version|latest] documentation for [library] and generate skills.
Library: [extracted library name]
Version: [extracted version or "latest"]
Custom URL: [if provided, otherwise omit]
After fetching:
1. Generate the expert skill using /doc-fetcher:generate-skill
2. Report the results including number of pages cached and skills generated
Step 3: Report Results
After the agent completes, summarize the results to the user:
- Number of pages cached
- Location of cached documentation
- Skills that were generated
- Any errors or warnings
Example Prompts That Activate This Skill
| User Says | Parsed As |
|---|---|
| "Get the latest documentation for Next.js" | library: nextjs, version: latest |
| "Fetch React 19 docs" | library: react, version: 19 |
| "I need the Tailwind CSS documentation" | library: tailwindcss, version: latest |
| "Download documentation for Prisma" | library: prisma, version: latest |
| "Update my Vue.js documentation" | library: vue, action: update |
| "Get docs from https://tanstack.com/query" | library: tanstack-query, url: provided |
Spawning the Agent
When spawning the doc-crawler agent, use this pattern:
Task tool with subagent_type: "doc-fetcher:doc-crawler"
Prompt: "Fetch the [version] documentation for [library].
Instructions:
1. Find the documentation URL (use resolver script or web search)
2. Run /doc-fetcher:fetch-docs [library] [version] [--url if provided]
3. After fetching, run /doc-fetcher:generate-skill [library] [version]
4. Report:
- Pages cached
- Cache location
- Skills generated
- Any errors or warnings"
Handling Special Cases
User Provides a URL
If the user provides a URL like "Fetch docs from https://example.com/docs":
- Extract a library name from the URL or ask the user
- Pass the URL to the agent with
--urlflag
Update Request
If the user says "update" instead of "fetch":
- Instruct the agent to use
/doc-fetcher:update-docsinstead - Regenerate skills after update
Multiple Libraries
If the user requests multiple libraries:
- Spawn separate agents for each library
- Summarize all results at the end
Version Not Specified
If no version is specified:
- Default to "latest"
- The fetch command will detect the current version
What NOT to Do
- DO NOT fetch documentation yourself using WebFetch
- DO NOT write custom crawling code
- DO NOT skip the doc-crawler agent - it handles all the complexity
- DO NOT forget to generate skills after fetching
Success Criteria
The task is complete when:
- Documentation is cached to
~/.claude/docs/[library]/[version]/ - At least one skill is generated (typically
[library]-[version]-expert) - User receives a clear summary of what was done
Documentation Indexer
I orchestrate the crawling, parsing, and indexing of documentation from web sources. I'm the core engine behind the /fetch-docs command.
What I Do
I handle the complete documentation fetching workflow:
- Discovery: Find documentation pages via sitemap.xml or navigation
- Framework Detection: Identify the documentation framework in use
- Crawling: Systematically fetch all documentation pages
- Content Extraction: Extract clean content from HTML (remove nav, footer, etc.)
- Storage: Organize and save documentation locally
- Indexing: Create searchable index and metadata
- Validation: Ensure completeness and quality
Workflow
Phase 1: Discovery
First, I determine how to discover all documentation pages:
1. Check for llms.txt (delegate to llms-txt-finder)
✗ Not found → Continue
2. Check for sitemap.xml
✓ Found at https://example.com/sitemap.xml
3. Parse sitemap
Found 234 URLs matching /docs/ pattern
4. Filter and validate URLs
Kept 234 documentation URLs
Excluded 45 blog/marketing URLs
Phase 2: Framework Detection
I detect the documentation framework to optimize extraction:
Supported Frameworks:
-
Docusaurus - Meta's React-based doc framework
- Detect:
docusaurus.config.jsor<meta name="generator" content="Docusaurus"> - Extract: Main article content, sidebar structure
- Navigate: Use generated sidebar for page discovery
- Detect:
-
VitePress - Vue-powered static site generator
- Detect:
.vitepress/config.jsor<meta name="generator" content="VitePress"> - Extract:
.vp-doccontent blocks - Navigate: Parse
config.jsfor sidebar structure
- Detect:
-
Nextra - Next.js-based docs framework (used by Next.js itself)
- Detect:
_meta.jsonfiles or Next.js patterns - Extract: Main content from MDX files
- Navigate: Follow
_meta.jsonhierarchy
- Detect:
-
GitBook - Popular documentation platform
- Detect:
SUMMARY.mdor GitBook meta tags - Extract: Main article content
- Navigate: Parse
SUMMARY.mdfor structure
- Detect:
-
Mintlify - Modern documentation platform (used by Supabase)
- Detect: Mintlify-specific meta tags
- Extract: Main content blocks
- Navigate: API-based navigation discovery
-
ReadTheDocs - Sphinx-based documentation
- Detect: ReadTheDocs theme or Sphinx meta
- Extract:
.documentor.bodycontent - Navigate:
searchindex.jsfor page list
-
Custom/Static - Fallback for custom sites
- Detect: No recognized framework
- Extract: Heuristic-based content extraction
- Navigate: Follow internal links carefully
Phase 3: Crawling
Systematic fetching with safeguards:
// Conceptual crawling algorithm
const crawlConfig = {
maxPages: 500, // Configurable limit
delayMs: 1000, // Rate limiting
timeout: 30000, // 30 second timeout per page
retries: 3, // Retry failed pages
respectRobotsTxt: true, // Honor robots.txt
userAgent: "Claude Code Doc Fetcher/1.0"
};
for (const url of discoveredUrls) {
// Rate limiting
await delay(crawlConfig.delayMs);
// Fetch with timeout
const html = await fetchWithTimeout(url, crawlConfig.timeout);
// Extract content
const content = await extractContent(html, framework);
// Save to cache
await saveToCache(url, content);
// Progress update
updateProgress(currentPage, totalPages);
}
Phase 4: Content Extraction
I extract clean, AI-friendly content:
What I Keep:
- Main article content
- Headings (h1-h6) with hierarchy
- Code blocks with language tags
- Lists and tables
- Images (with alt text)
- Inline code
- Blockquotes (notes, warnings, tips)
What I Remove:
- Navigation menus
- Headers and footers
- Sidebars
- Advertisement blocks
- Cookie notices
- Social media widgets
- JavaScript bundles
- Analytics scripts
Example Extraction:
Input (HTML):
<!DOCTYPE html>
<html>
<head>...</head>
<body>
<nav><!-- Navigation --></nav>
<main>
<h1>Server Actions</h1>
<p>Server Actions are asynchronous functions...</p>
<pre><code class="language-tsx">
export async function createUser(formData: FormData) {
'use server'
// ...
}
</code></pre>
</main>
<footer><!-- Footer --></footer>
</body>
</html>
Output (Markdown):
# Server Actions
Server Actions are asynchronous functions...
```tsx
export async function createUser(formData: FormData) {
'use server'
// ...
}
### Phase 5: Storage
I organize documentation in a structured format:
.claude/docs/ └── nextjs/ └── 15.0.3/ ├── index.json # Metadata ├── sitemap.json # Page hierarchy └── pages/ ├── getting-started.md ├── routing/ │ ├── introduction.md │ ├── defining-routes.md │ ├── pages-and-layouts.md │ └── ... ├── data-fetching/ │ ├── fetching.md │ ├── caching.md │ └── ... └── api-reference/ └── ...
**index.json** - Complete metadata:
```json
{
"library": "nextjs",
"version": "15.0.3",
"source_url": "https://nextjs.org/docs",
"fetched_at": "2025-01-17T10:30:00Z",
"framework": "nextra",
"page_count": 234,
"total_size_bytes": 5242880,
"llms_txt_url": null,
"sitemap_url": "https://nextjs.org/sitemap.xml",
"skill_generated": true,
"skill_path": "skills/nextjs-15-expert",
"extraction_strategy": "nextra-optimized",
"crawl_duration_seconds": 245,
"errors": 0,
"warnings": 3
}
sitemap.json - Page hierarchy:
{
"structure": [
{
"title": "Getting Started",
"path": "getting-started.md",
"children": []
},
{
"title": "Routing",
"path": "routing/",
"children": [
{
"title": "Introduction",
"path": "routing/introduction.md"
},
{
"title": "Defining Routes",
"path": "routing/defining-routes.md"
}
]
}
]
}
Phase 6: Indexing
Create searchable index for fast lookups:
{
"index_version": "1.0",
"created_at": "2025-01-17T10:35:00Z",
"pages": [
{
"path": "routing/defining-routes.md",
"title": "Defining Routes",
"headings": [
"Creating Routes",
"File Conventions",
"Dynamic Routes"
],
"keywords": ["routing", "file-system", "app-router", "dynamic"],
"code_languages": ["tsx", "jsx"],
"word_count": 1234
}
]
}
Framework-Specific Extraction
Docusaurus
Strategy:
1. Parse docusaurus.config.js for sidebar structure
2. Extract from <article> or .markdown elements
3. Preserve admonitions (:::note, :::tip, etc.)
4. Handle versioned docs (if multiple versions)
Example:
<article class="markdown">
<h1>Title</h1>
<p>Content...</p>
</article>
→
# Title
Content...
VitePress
Strategy:
1. Parse .vitepress/config.js for navigation
2. Extract from .vp-doc or .content divs
3. Handle frontmatter in markdown source
4. Preserve custom containers
Example selectors:
- .vp-doc
- .content
- main article
Nextra
Strategy:
1. Parse _meta.json files for structure
2. Extract main content from MDX
3. Handle Next.js-specific components
4. Follow nested _meta.json for hierarchy
Example:
_meta.json:
{
"index": "Introduction",
"routing": "Routing",
"data-fetching": "Data Fetching"
}
GitBook
Strategy:
1. Parse SUMMARY.md for table of contents
2. Extract from .page-inner or article elements
3. Handle GitBook-specific markdown extensions
4. Follow chapter structure
Example SUMMARY.md:
# Summary
* [Introduction](README.md)
* [Chapter 1](chapter1.md)
* [Section 1.1](chapter1/section1.md)
Mintlify
Strategy:
1. Use Mintlify API if available
2. Extract from main content containers
3. Handle MDX components
4. Parse mint.json for navigation
Example selectors:
- .docs-content
- main article
ReadTheDocs
Strategy:
1. Parse searchindex.js for page list
2. Extract from .document or .body
3. Handle Sphinx directives
4. Preserve code-block languages
Example selectors:
- div.document
- div.body
- section[role="main"]
Crawling Safeguards
Rate Limiting
Default: 1 second between requests
Configurable: 100ms - 5000ms
Behavior:
- Respect server delays
- Exponential backoff on errors
- Throttle on rate limit responses (429)
Robots.txt Compliance
1. Fetch robots.txt before crawling
2. Parse User-agent rules
3. Respect Disallow directives
4. Honor Crawl-delay if specified
Example robots.txt:
User-agent: *
Disallow: /admin
Crawl-delay: 1
Error Handling
404 Not Found:
→ Skip page, log warning, continue
500 Server Error:
→ Retry with exponential backoff (3 attempts)
Rate Limit (429):
→ Increase delay, retry after specified time
Timeout:
→ Retry with longer timeout
Network Error:
→ Retry, offer to pause/resume
Progress Reporting
Fetching Next.js documentation...
[████████████░░░░░░░░] 156/234 pages (67%)
Current: /docs/app/building-your-application/routing/defining-routes
Speed: 2.5 pages/sec
Elapsed: 1m 2s
Remaining: ~30s
Errors: 0
Content Validation
After crawling, I validate the content:
✓ All pages fetched (234/234)
✓ No missing dependencies
✓ Code blocks properly formatted
✓ Images referenced correctly
✓ Internal links valid
⚠ 3 external links broken (logged)
Quality Score: 98/100
Incremental Updates
When updating existing documentation:
1. Compare current index with new sitemap
2. Identify:
- New pages (added)
- Modified pages (changed)
- Removed pages (deleted)
3. Fetch only changed pages
4. Update index
5. Preserve unchanged content
Example:
Checking for changes...
New: 5 pages
Modified: 12 pages
Removed: 2 pages
Unchanged: 215 pages
Fetching only 17 changed pages...
(Saves ~90% of time vs full re-crawl)
Configuration
Configure my behavior in doc-fetcher-config.json:
{
"indexer": {
"crawl_delay_ms": 1000,
"max_pages_per_fetch": 500,
"timeout_ms": 30000,
"max_retries": 3,
"respect_robots_txt": true,
"user_agent": "Claude Code Doc Fetcher/1.0",
"frameworks": {
"auto_detect": true,
"priority": ["docusaurus", "vitepress", "nextra", "gitbook"]
},
"extraction": {
"remove_navigation": true,
"preserve_code_blocks": true,
"extract_images": true,
"follow_redirects": true
},
"validation": {
"check_links": true,
"validate_code": false,
"min_content_length": 100
}
}
}
Technical Implementation
I leverage these tools:
- WebFetch: Fetch HTML content from URLs
- Bash: Run curl commands for headers, robots.txt checking
- Read/Write: Manage local cache storage
- Glob: Find and organize cached files
- Task: Launch doc-crawler agent for complex sites
Example Crawl Session
/fetch-docs nextjs
→ Invoking doc-indexer skill...
[1/7] Discovery
✓ Checked for llms.txt (not found)
✓ Found sitemap.xml
✓ Parsed 234 documentation URLs
[2/7] Framework Detection
✓ Detected: Nextra
✓ Optimized extraction strategy loaded
[3/7] Robots.txt Check
✓ Fetched robots.txt
✓ No restrictions for our user-agent
✓ Crawl-delay: 1 second (using default)
[4/7] Crawling
[████████████████████] 234/234 (100%)
Duration: 4m 12s
Errors: 0
[5/7] Content Extraction
✓ Extracted clean markdown from all pages
✓ Preserved 156 code blocks
✓ Removed navigation/footer from all pages
[6/7] Storage
✓ Saved to .claude/docs/nextjs/15.0.3/
✓ Created index.json
✓ Created sitemap.json
Total size: 5.2 MB
[7/7] Validation
✓ All pages valid
✓ Code blocks properly formatted
✓ Links checked (3 external broken - logged)
Quality: 98/100
✓ Documentation indexed successfully!
Next: Generating skill (auto-enabled)
Run: /generate-doc-skill nextjs
Troubleshooting
"Framework detection failed"
- Falls back to generic extraction
- May be less accurate
- Consider manual framework hint
"Too many pages (>500)"
- Increase max_pages_per_fetch in config
- Or fetch specific sections manually
"Rate limited by server"
- Automatic: I increase delay automatically
- Manual: Increase crawl_delay_ms in config
"Content extraction poor quality"
- Framework detection may be wrong
- Custom site may need doc-crawler agent
- Check extraction selectors
See Also
llms-txt-finderskill - Checks for AI-optimized docs firstdoc-crawleragent - Advanced crawling for difficult sites/fetch-docscommand - Main entry point/update-docscommand - Incremental updates
Documentation Skill Generator
I generate Claude Code skills from cached documentation, creating version-specific expert skills that reference your locally cached docs.
What I Do
I transform cached documentation into functional Claude Code skills:
- Analyze Documentation: Read cached docs and extract structure
- Generate Skill Content: Create skill description and capabilities
- Configure Auto-Activation: Set up smart activation based on project context
- Create Metadata: Link skill to cached documentation
- Optimize for AI: Structure content for effective AI assistance
Skill Templates
I support multiple templates for different use cases:
1. Expert Template (Default)
Best for: Comprehensive library coverage
Features:
- Complete topic coverage
- All API references included
- Code examples from docs
- Auto-activation rules
- Version-specific guidance
- Migration notes (if available)
Use case:
"I want Claude to be an expert in Next.js 15"
2. Quick Reference Template
Best for: Fast lookups and common patterns
Features:
- Condensed documentation
- Top 20% most-used features
- Cheat-sheet style
- Minimal context overhead
- Fast activation
Use case:
"I just need quick API references"
3. Migration Guide Template
Best for: Version upgrades
Features:
- What's new comparison
- Breaking changes highlighted
- Before/after examples
- Deprecation warnings
- Upgrade checklist
Use case:
"I'm upgrading from Next.js 14 to 15"
Requirements:
- Multiple versions cached
4. Troubleshooter Template
Best for: Debugging and problem-solving
Features:
- Common errors catalog
- Debug strategies
- Performance issues
- Configuration problems
- Stack trace interpretation
Use case:
"Help me debug issues faster"
5. Best Practices Template
Best for: Code quality and optimization
Features:
- Recommended patterns
- Anti-patterns to avoid
- Performance optimization
- Security considerations
- Accessibility tips
Use case:
"Ensure I'm using the library correctly"
Skill Generation Process
Step 1: Load Documentation
Reading cached documentation...
Path: .claude/docs/nextjs/15.0.3/
Files: 234 markdown files
Size: 5.2 MB
Index: index.json loaded
Sitemap: sitemap.json loaded
Step 2: Analyze Content
// Conceptual analysis
const analysis = {
topics: extractTopics(pages), // ["Routing", "Data Fetching", ...]
apiMethods: extractAPIMethods(pages), // ["fetch", "cookies", "headers", ...]
codeExamples: extractCodeBlocks(pages), // 156 code examples
sections: buildHierarchy(sitemap), // Nested structure
keywords: generateKeywords(content), // ["server actions", "app router", ...]
version: extractVersion(metadata) // "15.0.3"
};
Step 3: Generate Skill Content
Based on the selected template, I create the skill file:
Example: Expert Template
---
name: nextjs-15-expert
description: Expert knowledge of Next.js 15.0.3 based on official documentation
version: 15.0.3
library: nextjs
docs_path: .claude/docs/nextjs/15.0.3
auto_activate: true
activation_patterns:
- "package.json contains next@15"
- "directory contains app/ or pages/"
- "imports from 'next'"
---
# Next.js 15 Expert
I have comprehensive, version-specific knowledge of Next.js 15.0.3 based on the complete official documentation.
## Topics I Cover
{/* Auto-generated from documentation structure */}
- **Routing** (App Router & Pages Router)
- **Data Fetching** (Server Components, fetch API, caching)
- **Rendering** (SSG, SSR, CSR, Streaming)
- **Server Actions** (Mutations, form handling)
- **API Routes** (Route handlers, middleware)
- **Optimization** (Images, fonts, scripts)
- **Metadata** (SEO, Open Graph)
- **Configuration** (next.config.js options)
## What I Know
{/* Auto-generated capability list */}
I can help you with:
- Setting up new Next.js 15 projects
- Implementing App Router patterns
- Creating Server Actions for mutations
- Optimizing performance and bundle size
- Configuring metadata for SEO
- Deploying to Vercel or self-hosting
- Migrating from Pages Router to App Router
- Troubleshooting common issues
## Code Examples
{/* Reference to extracted code examples */}
I have access to 156 code examples covering:
- TypeScript and JavaScript
- Server Components
- Client Components
- Server Actions
- Route Handlers
- Middleware
- Configuration
## Version Notes
This skill is specifically for Next.js **15.0.3**.
Key features in this version:
- Partial Prerendering (PPR)
- Enhanced Server Actions
- Improved fetch caching
- Better error handling
I will NOT suggest deprecated patterns from earlier versions.
---
*Generated by doc-fetcher plugin*
*Source: https://nextjs.org/docs*
*Cached: 2025-01-17*
Step 4: Create Metadata
I create a .doc-reference.json file to link the skill to cached docs:
{
"library": "nextjs",
"version": "15.0.3",
"docs_path": ".claude/docs/nextjs/15.0.3",
"generated_at": "2025-01-17T15:30:00Z",
"template_used": "expert",
"source_url": "https://nextjs.org/docs",
"page_count": 234,
"code_examples": 156,
"auto_generated": true,
"last_regenerated": "2025-01-17T15:30:00Z"
}
Step 5: Configure Auto-Activation
I set up intelligent activation rules:
# When to auto-activate this skill
activation_patterns:
# package.json dependency
- "package.json contains next@15"
- "package.json contains next@^15"
# Directory structure
- "directory contains app/"
- "directory contains pages/"
# File imports
- "imports from 'next'"
- "imports from 'next/server'"
- "imports from 'next/navigation'"
# File names
- "file named next.config.js"
- "file named middleware.ts"
- "file matches app/**/*.tsx"
# User intent
- "user asks about Next.js 15"
- "user mentions App Router"
- "user mentions Server Actions"
activation_priority: high # high, medium, low
activation_confidence: 0.9 # 0.0 - 1.0
Template Customization
Expert Template
Includes:
- All topics from documentation
- Complete API reference
- All code examples
- Migration notes
- Best practices sections
Content structure:
1. Overview
2. Topics covered (exhaustive)
3. What I know (capabilities)
4. Code examples catalog
5. Version-specific notes
6. Usage examples
Activation:
- Aggressive auto-activation
- High confidence threshold
Quick Reference Template
Includes:
- Top 20% most-used features only
- Essential APIs
- Common code patterns
- Quick tips
Content structure:
1. Brief overview
2. Core features (condensed)
3. Common patterns
4. Quick reference tables
Activation:
- Manual activation preferred
- Low overhead
Migration Guide Template
Includes:
- Version comparison (e.g., 14 → 15)
- Breaking changes
- New features
- Deprecated features
- Code migration examples
Content structure:
1. What's new
2. Breaking changes
3. Deprecated APIs
4. Migration checklist
5. Before/after examples
Activation:
- On version mismatch detection
- Manual for upgrade planning
Requirements:
- Requires multiple versions cached
- Compares v14 vs v15 docs
Troubleshooter Template
Includes:
- Common errors from docs
- Debug strategies
- Performance issues
- Configuration gotchas
Content structure:
1. Error catalog
2. Debugging tips
3. Performance checklist
4. Configuration guide
5. FAQ
Activation:
- On error keywords
- Manual for debugging sessions
Best Practices Template
Includes:
- Recommended patterns from docs
- Anti-patterns
- Performance tips
- Security guidelines
Content structure:
1. Do's and Don'ts
2. Performance optimization
3. Security best practices
4. Accessibility considerations
5. Production checklist
Activation:
- Manual for code review
- On "best practice" queries
Content Extraction Intelligence
I intelligently extract different content types:
API Methods
// Detected from documentation
const apiMethods = [
{
name: "cookies()",
signature: "cookies(): RequestCookies",
description: "Read incoming request cookies",
availability: "Server Components, Route Handlers",
example: "const cookieStore = cookies(); const theme = cookieStore.get('theme')"
},
// ... more methods
];
Code Examples
// Extracted and categorized
const codeExamples = [
{
language: "tsx",
category: "Server Actions",
title: "Creating a Server Action",
code: "export async function createUser(formData: FormData) { ... }",
sourceUrl: "https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions"
},
// ... 155 more examples
];
Topics and Sections
// Built from sitemap hierarchy
const topics = [
{
title: "Routing",
pages: 24,
subtopics: [
"Defining Routes",
"Pages and Layouts",
"Dynamic Routes",
"Route Groups",
"Parallel Routes"
]
},
// ... more topics
];
Skill Naming
I generate meaningful skill names:
// Naming patterns
const patterns = {
expert: "{library}-{major-version}-expert",
quickRef: "{library}-quick-ref",
migration: "{library}-{old-version}-to-{new-version}",
troubleshoot: "{library}-troubleshooter",
bestPractices: "{library}-best-practices"
};
// Examples:
// "nextjs-15-expert"
// "supabase-quick-ref"
// "react-17-to-18"
// "nextjs-troubleshooter"
// "nextjs-best-practices"
Regeneration
When documentation is updated, I intelligently regenerate:
Detected documentation update for Next.js
Old: 15.0.0
New: 15.0.3
Regenerating skill: nextjs-15-expert
Changes detected:
- 12 new pages added
- 3 pages modified
- 1 page removed
- 8 new code examples
Skill updated:
✓ New features added to topics
✓ Code examples refreshed
✓ Version metadata updated
✓ Activation rules preserved
Skill ready: nextjs-15-expert
Output Example
Generating skill for Next.js 15.0.3...
[1/5] Loading documentation
✓ Read 234 markdown files
✓ Loaded metadata and sitemap
✓ Parsed code examples
[2/5] Analyzing content
✓ Identified 12 major topics
✓ Extracted 156 code examples
✓ Found 89 API methods
✓ Generated 247 keywords
[3/5] Generating skill (expert template)
✓ Created skill description
✓ Listed all topics and capabilities
✓ Included code example references
✓ Added version-specific notes
[4/5] Configuring auto-activation
✓ Set activation patterns
✓ Priority: high
✓ Confidence: 0.9
[5/5] Saving skill
✓ Written to: skills/nextjs-15-expert/SKILL.md
✓ Created metadata: .doc-reference.json
✓ Registered with Claude Code
✓ Skill generated successfully!
Skill name: nextjs-15-expert
Location: skills/nextjs-15-expert/
Template: expert
Auto-activation: enabled
The skill is now active. Ask me anything about Next.js 15!
Configuration
Configure my behavior in doc-fetcher-config.json:
{
"skill_generation": {
"auto_generate": true,
"default_template": "expert",
"auto_activate": true,
"naming_pattern": "{library}-{major-version}-expert",
"output_directory": "skills",
"include_code_examples": true,
"max_examples_per_topic": 5,
"generate_metadata": true
}
}
Advanced Features
Multi-Version Skills
Generate and maintain skills for multiple versions:
/generate-doc-skill nextjs 14.2.0 --output skills/nextjs-14-expert
/generate-doc-skill nextjs 15.0.3 --output skills/nextjs-15-expert
Both skills coexist. Activation rules ensure the correct version is used.
Skill Merging
Combine multiple libraries into one skill:
/generate-doc-skill nextjs,react,typescript --template full-stack
Creates a unified skill with knowledge of all three libraries.
Custom Templates
Create your own templates:
# templates/my-custom-template.yml
name: "{library}-custom"
description: "Custom skill for {library}"
sections:
- overview
- quick_start
- api_reference
activation: manual
Use:
/generate-doc-skill nextjs --template ./templates/my-custom-template.yml
Troubleshooting
"Skill generation failed"
- Check documentation cache exists
- Verify index.json is valid
- Re-fetch docs if corrupted
"Skill too large"
- Use
quick-referencetemplate instead - Limit code examples in config
- Filter to specific topics
"Skill not auto-activating"
- Check activation patterns in SKILL.md
- Verify package.json contains dependency
- Manually activate first time
"Outdated skill content"
- Run
/update-docs <library> - Regenerate with
/generate-doc-skill <library>
See Also
/generate-doc-skillcommand - Main entry pointdoc-indexerskill - Creates the cached docs I use/fetch-docscommand - Auto-generates skills after fetching
LLMs.txt Finder
I specialize in detecting and fetching AI-optimized documentation formats like llms.txt and claude.txt from documentation websites.
What I Do
I help the doc-fetcher plugin find the most AI-friendly documentation format available:
- Check for llms.txt: Look for
/llms.txtor/llms-full.txtat the site root - Check for claude.txt: Look for Claude-specific documentation at
/claude.txt - Parse and validate: Ensure the file contains valid documentation content
- Extract metadata: Pull version info, update dates, and structure from the file
- Download content: Fetch the complete content for caching
AI-Optimized Documentation Formats
llms.txt
A standard format designed specifically for AI consumption:
# Library Name
> Version: 1.0.0
> Last Updated: 2025-01-17
## Overview
[Concise description optimized for AI understanding]
## Installation
[Installation instructions]
## Quick Start
[Getting started guide]
## API Reference
[Complete API documentation]
## Examples
[Code examples with context]
Benefits:
- Pre-formatted for AI parsing
- Optimized for token efficiency
- Includes structured metadata
- Often more concise than web docs
claude.txt
Claude-specific documentation format:
# Library Name for Claude
This documentation is optimized for Claude AI assistant.
[Documentation structured specifically for Claude's capabilities]
Detection Strategy
When given a documentation URL, I check in this order:
https://example.com/llms-full.txt(comprehensive version)https://example.com/llms.txt(standard version)https://example.com/claude.txt(Claude-specific)https://example.com/.well-known/llms.txt(alternative location)https://example.com/docs/llms.txt(docs subdirectory)
Usage
Automatic Activation
I'm automatically invoked when you run:
/fetch-docs <library>
I run first and check for AI-optimized formats before falling back to web crawling.
Manual Usage
You can also invoke me directly through the skill:
"Check if Next.js has an llms.txt file"
"Find AI-optimized docs for Supabase"
"Does https://react.dev have claude.txt?"
What I Return
If I find an AI-optimized documentation file, I return:
{
"found": true,
"type": "llms.txt",
"url": "https://nextjs.org/llms.txt",
"size_bytes": 524288,
"version": "15.0.3",
"last_updated": "2025-01-15",
"content_preview": "# Next.js 15.0.3...",
"should_use": true,
"reason": "AI-optimized format available, more efficient than crawling"
}
If not found:
{
"found": false,
"checked_urls": [
"https://example.com/llms.txt",
"https://example.com/claude.txt",
"https://example.com/.well-known/llms.txt"
],
"fallback": "sitemap.xml",
"reason": "No AI-optimized format found, falling back to web crawling"
}
Integration with Doc Fetcher
When I find an AI-optimized file:
- Skip Crawling: No need to crawl hundreds of pages
- Faster Fetching: Single file download vs. many HTTP requests
- Better Structure: Pre-formatted for AI understanding
- Token Efficient: Content is already optimized for LLM context windows
- More Accurate: Maintained specifically for AI assistants
Example Flow
User: /fetch-docs nextjs
Doc Fetcher: Invoking llms-txt-finder skill...
llms-txt-finder:
✓ Checking https://nextjs.org/llms.txt
✓ Found! (524 KB)
✓ Version: 15.0.3
✓ Last updated: 2025-01-15
Recommendation: Use llms.txt instead of crawling
Benefits:
- 1 file vs ~234 pages to crawl
- Pre-optimized for AI
- Faster download (5 seconds vs 4 minutes)
Doc Fetcher: Using llms.txt...
✓ Downloaded and cached
✓ Generated skill: nextjs-15-expert
Validation
I validate AI-optimized documentation files for:
- Format: Proper markdown structure
- Completeness: Contains essential sections (installation, API, examples)
- Version Info: Includes version metadata
- Size: Reasonable size (not truncated or too small)
- Content Quality: Actual documentation, not just a placeholder
If validation fails, I recommend falling back to crawling:
⚠ Found llms.txt but validation failed
Issue: File is only 2 KB (likely incomplete)
Recommendation: Fall back to sitemap.xml crawling
Known Sites with AI-Optimized Docs
I maintain awareness of popular libraries that provide AI-optimized documentation:
- Next.js:
/llms.txt✓ - Vercel AI SDK:
/llms.txt✓ - Anthropic Claude:
/claude.txt✓ - OpenAI:
/llms.txt✓ - Supabase: Checking...
- Tailwind CSS: Checking...
This list helps me make intelligent guesses about where to look.
Creating llms.txt for Your Project
If you maintain a library, I can help you create an llms.txt file:
"Help me create an llms.txt file for my library"
I'll generate a template following best practices:
# Your Library Name
> Version: 1.0.0
> Last Updated: 2025-01-17
> Repository: https://github.com/you/your-lib
> Documentation: https://docs.yourlib.com
## Overview
[2-3 sentence description of what your library does]
## Installation
[Installation commands for different package managers]
## Quick Start
[Minimal example to get started]
## Core Concepts
[Key concepts users need to understand]
## API Reference
[Complete API documentation]
## Examples
[Common use cases with code examples]
## Advanced Usage
[Advanced patterns and techniques]
## Troubleshooting
[Common issues and solutions]
## Changelog
[Recent changes and migration notes]
Performance Impact
Using AI-optimized documentation dramatically improves performance:
| Metric | Web Crawling | llms.txt |
|---|---|---|
| Files fetched | 100-500 | 1 |
| Time to fetch | 2-10 min | 5-30 sec |
| Network requests | 100-500 | 1 |
| Size on disk | 5-50 MB | 0.5-5 MB |
| Token efficiency | Variable | Optimized |
Error Handling
I handle various error scenarios:
- 404: File doesn't exist, fall back to crawling
- 403: Access forbidden, try alternative URLs
- Timeout: Server slow, retry with backoff
- Invalid format: Not proper documentation, fall back
- Too large: File >50MB, might be data dump, validate carefully
Configuration
Configure my behavior in doc-fetcher-config.json:
{
"llms_txt": {
"enabled": true,
"check_locations": [
"/llms-full.txt",
"/llms.txt",
"/claude.txt",
"/.well-known/llms.txt",
"/docs/llms.txt"
],
"max_size_bytes": 52428800,
"validation_strict": true,
"prefer_over_crawling": true
}
}
Future Enhancements
- Registry: Maintain a registry of known llms.txt URLs
- Validation Service: Validate llms.txt files against a schema
- Generator: Auto-generate llms.txt from existing docs
- Suggestions: Suggest improvements to existing llms.txt files
Technical Implementation
I use these tools to accomplish my tasks:
- WebFetch: Fetch content from URLs
- Bash: Check HTTP headers with curl
- Read: Read configuration files
- Write: Cache fetched llms.txt files
Example Checks
Successful Find
Checking for AI-optimized docs at https://nextjs.org...
✓ https://nextjs.org/llms.txt
Status: 200 OK
Size: 524 KB
Content-Type: text/plain
Last-Modified: 2025-01-15
✓ Validation passed
- Valid markdown format
- Contains version: 15.0.3
- Comprehensive content
- Reasonable size
Recommendation: Use llms.txt
Benefits: 40x faster than crawling 234 pages
Not Found
Checking for AI-optimized docs at https://example.com...
✗ https://example.com/llms-full.txt (404)
✗ https://example.com/llms.txt (404)
✗ https://example.com/claude.txt (404)
✗ https://example.com/.well-known/llms.txt (404)
No AI-optimized documentation found.
Falling back to sitemap.xml or web crawling.
See Also
doc-indexerskill - Main documentation crawling logicdoc-crawleragent - Advanced web crawling for non-standard sites/fetch-docscommand - Primary entry point that uses this skill
Commands 5
Doc-Fetcher Configuration
View and modify doc-fetcher plugin configuration settings.
Current Context
- Current working directory: !
pwd
Your Task
When the user invokes this command, follow these steps:
- Parse Arguments: The user provided:
$ARGUMENTS--showor no arguments: Display current configuration with explanations--set <key> <value>: Update a specific configuration value--reset: Reset configuration to defaults--path: Show the config file path--restore: Restore configuration from a backup (see step 6)
1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.
-
Run Config Command:
- Execute:
node ~/.claude/plugins/cache/doc-fetcher/scripts/config.js $ARGUMENTS - The script handles:
- Displaying current configuration with descriptions
- Validating and updating configuration values
- Resetting to default values
- Showing config file location
- Execute:
-
Handle Results:
- For
--show: Display a formatted table of all settings with current values and descriptions - For
--set: Confirm the change was made and show the new value - For
--reset: Confirm defaults were restored - For
--path: Show the full path to the config file
- For
-
Available Settings:
Setting Type Description crawl_delay_msnumber Delay between page fetches (ms). Increase if rate limited. max_pages_per_fetchnumber Maximum pages to fetch per library. auto_generate_skillsboolean Automatically generate skills after fetching docs. timeout_msnumber Request timeout in milliseconds. max_retriesnumber Number of retry attempts on network failure. enable_checkpointsboolean Enable resume for interrupted fetches. checkpoint_intervalnumber Pages between checkpoint saves. respect_robots_txtboolean Honor robots.txt crawl rules. fetch_llms_urlsboolean Fetch individual pages from llms.txt files. user_agentstring User agent string for HTTP requests. -
Error Handling:
- If invalid key: Show list of valid configuration keys
- If invalid value type: Show expected type and example
- If config file missing: Will be created with defaults
- If permission error: Suggest checking file permissions
-
Handle --restore (DO NOT run the node script for this):
- List backup files:
ls -la ~/.claude/doc-fetcher/ - If no backups exist, inform the user and exit
- Display the backups as a numbered list with timestamps (parse from filename format:
doc-fetcher-config.YYYY-MM-DDTHH-MM-SS-MMMZ.json) - Use AskUserQuestion to ask the user which backup to restore (show numbers 1, 2, 3, etc.)
- After user selection, copy the selected backup:
cp ~/.claude/doc-fetcher/<selected-backup-file> ~/.claude/plugins/cache/doc-fetcher/doc-fetcher-config.json - Confirm the restore was successful
- List backup files:
Examples
# Show all current settings
/doc-fetcher:config
# Show config with full details
/doc-fetcher:config --show
# Increase crawl delay to avoid rate limiting
/doc-fetcher:config --set crawl_delay_ms 2000
# Disable automatic skill generation
/doc-fetcher:config --set auto_generate_skills false
# Reset all settings to defaults
/doc-fetcher:config --reset
# Show config file location
/doc-fetcher:config --path
# Restore from a previous backup
/doc-fetcher:config --restore
Important Notes
- Configuration is stored in
~/.claude/plugins/cache/doc-fetcher/doc-fetcher-config.json - Automatic backups: Every config change creates a timestamped backup in
~/.claude/doc-fetcher/ - Use
--restoreto restore from any previous backup - Changes take effect immediately for new commands
- Boolean values: use
trueorfalse - Number values: integers only (no quotes needed)
- String values: can include spaces if quoted
Fetch Documentation
Fetch and cache documentation for a specified library, creating a local snapshot in ~/.claude/docs for accurate, version-specific AI context.
Current Context
- Current working directory: !
pwd
Your Task
When the user invokes this command, follow these steps:
- Parse Arguments: The user provided:
$ARGUMENTS- First argument is the library name (required)
- Second argument is optional version (e.g., "15.0.0")
- Optional
--urlflag for custom documentation URLs - If library is missing: prompt for the library name
1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.
-
Run Fetch Command:
- Execute:
node ~/.claude/plugins/cache/doc-fetcher/scripts/fetch-docs.js $ARGUMENTS - The script will:
- Discover documentation format (llms.txt, claude.txt, or sitemap.xml)
- Detect the documentation framework (Docusaurus, VitePress, Nextra, etc.)
- Crawl and cache pages to
~/.claude/docs/[library]/[version]/ - Respect robots.txt and rate limits
- Save metadata in
index.jsonand structure insitemap.json
- Monitor progress and show status to the user
- Execute:
-
Handle Results:
- If successful: Report:
- Number of pages fetched
- Storage location (
~/.claude/docs/[library]/[version]/) - Total size
- Whether a skill was auto-generated
- Show next steps (how to use the cached docs or generate a skill)
- If successful: Report:
-
Error Handling:
- If URL not found (404): Verify the library name or try
--urlflag with custom URL - If network errors: Suggest retrying or checking connection
- If rate limited: Suggest increasing
crawl_delay_msindoc-fetcher-config.json - If documentation too large (>500 pages): Inform user and ask if they want to continue
- If framework not detected: Suggest using
--urlwith the docs homepage - If script not found: Verify the doc-fetcher plugin is installed
- If URL not found (404): Verify the library name or try
-
Skill Generation: If
auto_generate_skills: truein config, inform the user that a skill was automatically created and how to use it.
Important Notes
- The script runs from the plugin directory:
~/.claude/plugins/cache/doc-fetcher/ - Documentation is cached globally in
~/.claude/docsand shared across all projects - The cache directory is configured in
doc-fetcher-config.jsonand defaults to~/.claude/docs - Fetching can take several minutes for large documentation sets
- Use specific versions (e.g.,
/fetch-docs nextjs 15.0.0) to match your project's package.json - The fetched docs are stored locally and won't be updated automatically (use
/update-docsfor that)
Generate Documentation Skill
Create a Claude Code skill from cached documentation in ~/.claude/docs, enabling version-specific expertise and accurate AI guidance. Skills are saved to ~/.claude/skills.
Current Context
- Current working directory: !
pwd
Your Task
When the user invokes this command, follow these steps:
- Parse Arguments: The user provided:
$ARGUMENTS- First argument is the library name (required)
- Second argument is optional version (defaults to latest cached version)
--template <name>: Skill template to use (default: "expert")--output <path>: Custom output path for the skill- If library is missing: prompt for the library name
1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.
-
Run Generate Skill Command:
- Execute:
node ~/.claude/plugins/cache/doc-fetcher/scripts/generate-skill.js $ARGUMENTS - The script will:
- Locate cached docs in
~/.claude/docs/[library]/[version]/ - Analyze the documentation structure and content
- Generate a skill file with version-specific expertise
- Save skill to
~/.claude/skills/[library]-[version]-[template]/ - Link the skill to the cached documentation
- Locate cached docs in
- Monitor progress and capture output
- Execute:
-
Handle Results:
- If successful: Report:
- Skill name and location
- Version the skill references
- Topics covered
- Auto-activation rules (if any)
- How to use the skill
- Show the generated skill path
- If successful: Report:
-
Error Handling:
- If documentation not found: Suggest
/fetch-docs [library]to cache docs first - If skill already exists: Ask if user wants to overwrite or create a variant
- If template not found: List available templates and ask user to choose
- If generation fails: Show error and suggest re-fetching documentation
- If script not found: Verify the doc-fetcher plugin is installed
- If documentation not found: Suggest
-
Next Steps:
- IMPORTANT: Tell the user they need to restart Claude Code (exit and reopen) to load the newly generated skills
- Explain that skills will automatically activate when working with the library once Claude is restarted
Important Notes
- The script runs from the plugin directory:
~/.claude/plugins/cache/doc-fetcher/ - Documentation is cached globally in
~/.claude/docsand skills in~/.claude/skills - Skills are available across all projects (global Claude Code skills)
- The generated skill references the specific version of documentation
- Skills can auto-activate based on package.json dependencies or file patterns
- Use
--templateto customize skill focus (expert, quick-reference, troubleshooter, etc.) - Multiple skills can reference the same cached documentation with different templates
- Regenerate skills after updating documentation to get the latest content
List Documentation
Display all cached documentation from ~/.claude/docs including versions, sizes, update times, and associated skills.
Current Context
- Current working directory: !
pwd
Your Task
When the user invokes this command, follow these steps:
- Parse Arguments: The user provided:
$ARGUMENTS--project: Show only docs relevant to current project's package.json dependencies--verbose: Show detailed information including page counts, sources, and full metadata- If no arguments: show default view (all cached docs)
1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.
-
Run List Command:
- Execute:
node ~/.claude/plugins/cache/doc-fetcher/scripts/list-docs.js $ARGUMENTS - The script will scan
~/.claude/docsand display:- Library names and versions
- Number of pages cached
- Storage sizes
- Last update times
- Associated skills (if any)
- Capture and display the output
- Execute:
-
Present Results:
- Show the formatted output from the list command
- If using
--projectflag:- Highlight which dependencies are cached and current
- Identify missing or outdated documentation
- Suggest fetch/update commands for missing/outdated docs
- If using
--verboseflag:- Show detailed metadata (source URLs, frameworks, page counts, local paths)
-
Provide Suggestions:
- If docs are outdated: Suggest
/update-docs [library] - If docs are missing for project dependencies: Suggest
/fetch-docs [library] - If old versions exist: Mention cleanup options
- If docs are outdated: Suggest
-
Error Handling:
- If cache directory doesn't exist: Inform user no docs are cached yet, suggest fetching docs
- If cache directory is empty: Suggest using
/fetch-docsto start caching documentation - If package.json not found (when using
--project): Inform user and fall back to showing all cached docs - If script not found: Verify the doc-fetcher plugin is installed
Important Notes
- The script runs from the plugin directory:
~/.claude/plugins/cache/doc-fetcher/ - Documentation is cached globally in
~/.claude/docsand shared across all projects - The list command is fast (reads index files only, doesn't scan page content)
- The cache directory is configured in
doc-fetcher-config.jsonand defaults to~/.claude/docs - Use
--projectto see which of your current project's dependencies have cached documentation - Use
--verbosefor complete details including source URLs and local paths
Update Documentation
Check for and update outdated documentation in ~/.claude/docs.
Current Context
- Current working directory: !
pwd
Your Task
When the user invokes this command, follow these steps:
- Parse Arguments: The user provided:
$ARGUMENTS- If no arguments: prompt user to specify a library, use
--all, or use--project - Valid flags:
--all(update all cached docs),--project(update project dependencies),--force(force re-fetch)
- If no arguments: prompt user to specify a library, use
1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.
-
Run Update Command:
- Execute:
node ~/.claude/plugins/cache/doc-fetcher/scripts/update-docs.js $ARGUMENTS - It will check
~/.claude/docsfor cached documentation - Wait for completion and capture the output
- Execute:
-
Handle Results:
- If successful: Report what was updated (libraries, versions, page counts)
- If no updates needed: Inform the user their docs are current
- If errors occur: Show the error and suggest next steps
-
Error Handling:
- If documentation not found: Suggest using
/fetch-docsfirst - If network errors: Suggest retrying or checking connection
- If rate limited: Suggest waiting or increasing
crawl_delay_msin config - If script not found: Verify the doc-fetcher plugin is installed
- If documentation not found: Suggest using
-
Next Steps: If updates were made and skills exist, remind the user that associated skills have been automatically regenerated.
Important Notes
- The script runs from the plugin directory:
~/.claude/plugins/cache/doc-fetcher/ - Documentation is cached globally in
~/.claude/docsand shared across all projects - The cache directory is configured in
doc-fetcher-config.jsonand defaults to~/.claude/docs - Updates check for newer versions and re-fetch if available
- Use
--forceto re-fetch even if recently updated - The
--projectflag reads the current project's dependencies and updates matching docs in the global cache
Agents 1
Documentation Crawler Agent
Your Mission
You are the orchestration agent for the doc-fetcher plugin. Your job is to:
- Find the correct documentation URL for the requested library
- Fetch documentation using the plugin's slash commands
- Generate skills from the cached documentation
- Report results to the user
CRITICAL RULES:
- Use
/doc-fetcher:fetch-docsfor fetching documentation - Use
/doc-fetcher:generate-skillfor generating skills - Use
/doc-fetcher:list-docsto check cached documentation - Use
/doc-fetcher:update-docsto update existing documentation - NEVER create custom crawler files or write crawling logic
- MONITOR command output and report progress to the user
- HANDLE errors by adjusting parameters and retrying
Step 1: Finding the Documentation URL
When a user asks you to fetch documentation for a library, you need to find the correct URL. Use these strategies in order:
Strategy A: Check if URL is Provided
If the user provides a URL directly, use it:
User: "Fetch docs from https://nextjs.org/docs"
→ Use: /doc-fetcher:fetch-docs nextjs --url https://nextjs.org/docs
Strategy B: Use the URL Resolver Script
For most libraries, the resolve-docs-url.js script is the best option. It:
- Queries the SquirrelSoft API (centralized docs URL cache)
- Checks package registries (npm, PyPI, crates.io)
- Looks for llms.txt files
- Validates URLs before returning
node scripts/resolve-docs-url.js <library-name>
Example output:
🔍 Resolving documentation URL for "axios"...
Ecosystem: npm
Checking SquirrelSoft registry...
✓ Found in SquirrelSoft: https://axios-http.com/docs
✓ Resolved: https://axios-http.com/docs (source: squirrelsoft)
Strategy C: Web Search
If the resolver script fails or returns no results, use WebSearch to find the documentation:
Search queries to try:
"[library] official documentation""[library] docs site""[library] API reference"
What to look for in search results:
- Official documentation sites (usually
[library].dev,[library].io,docs.[library].com) - Avoid GitHub README pages (prefer dedicated doc sites)
- Avoid third-party tutorials or blog posts
- Look for sites with
/docs,/documentation,/api,/guide,/referencepaths
Strategy D: Common URL Patterns
If all else fails, try these common patterns:
https://[library].dev/docshttps://[library].io/docshttps://docs.[library].comhttps://[library].org/docshttps://[library]js.org(for JS libraries)
When to Use Each Strategy
| Scenario | Best Strategy |
|---|---|
| User provides URL | Use it directly (Strategy A) |
| Well-known library (react, vue, nextjs) | Resolver script (Strategy B) |
| Library not in registries | Web search (Strategy C) |
| Resolver returns GitHub URL | Try web search for better docs |
| New or obscure library | Combine web search + URL patterns |
Step 2: Fetching Documentation
Once you have the documentation URL, use the fetch command:
Basic Fetch (library name only)
/doc-fetcher:fetch-docs <library>
The command will auto-resolve the URL using the resolver script.
Fetch with Specific Version
/doc-fetcher:fetch-docs <library> <version>
Fetch with Custom URL
/doc-fetcher:fetch-docs <library> --url <custom-url>
Fetch with Version and Custom URL
/doc-fetcher:fetch-docs <library> <version> --url <custom-url>
Step 3: Monitoring Output
The fetch command outputs progress in 7 steps:
[1/7] Checking robots.txt...
[2/7] Checking for AI-optimized documentation...
[3/7] Parsing sitemap.xml...
[4/7] Crawling documentation pages...
[5/7] Saving to cache...
[6/7] Saving metadata...
[7/7] Summary
Watch for:
- ✓ Success messages
- ⚠ Warnings (skipped pages, robots.txt disallow, etc.)
- ✗ Errors (404s, network issues, parsing failures)
Step 4: Generating Skills
After documentation is fetched, generate skills:
Generate Default Expert Skill
/doc-fetcher:generate-skill <library>
Generate Specific Version Skill
/doc-fetcher:generate-skill <library> <version>
Generate with Specific Template
/doc-fetcher:generate-skill <library> --template <template-name>
Available Templates:
| Template | Purpose |
|---|---|
expert | Comprehensive coverage (default) |
quick-reference | Top 20% most-used features |
migration-guide | Version upgrade guidance |
troubleshooter | Common errors and debugging |
best-practices | Recommended patterns |
Step 5: Reporting to User
Summarize the result:
Success:
✓ Successfully fetched [library] v[version] documentation
- Cached [X] pages to ~/.claude/docs/[library]/[version]/
- Generated skill: [library]-[version]-expert
Partial Success:
⚠ Fetched [library] documentation with warnings
- Cached [X] pages ([Y] skipped)
- Reason: [explain skipped pages]
Failure:
✗ Could not fetch documentation for [library]
- Error: [explain error]
- Suggestion: [provide actionable fix]
Step 6: Handling Errors
| Error Type | Solution |
|---|---|
| URL not found | Try web search to find correct URL |
| No sitemap found | Provide explicit URL with --url flag |
| Robots.txt blocks | Inform user - cannot crawl |
| Network timeouts | Suggest retry - server may be slow |
| 404 errors | URL may be wrong - try alternative |
| Rate limiting | Wait and retry (handled automatically) |
Example Workflows
Example 1: Standard Documentation Fetch
User: "Get the Next.js 15 documentation"
Your actions:
- Run resolver to find URL:
node scripts/resolve-docs-url.js nextjs - Execute fetch command:
/doc-fetcher:fetch-docs nextjs 15 - Monitor output and report results
- Generate skills:
/doc-fetcher:generate-skill nextjs 15
Report to user:
"✓ Successfully fetched Next.js 15 documentation. Cached 234 pages. Generated nextjs-15-expert skill."
Example 2: Unknown Library
User: "Fetch documentation for some-new-lib"
Your actions:
- Try resolver script:
node scripts/resolve-docs-url.js some-new-lib - If no result, use WebSearch:
Search: "some-new-lib official documentation" - Find the docs URL from search results
- Fetch with explicit URL:
/doc-fetcher:fetch-docs some-new-lib --url https://some-new-lib.dev/docs
Example 3: Custom URL Provided
User: "Fetch docs from https://tanstack.com/query/latest/docs"
Your actions:
- Parse library name from URL (tanstack-query)
- Execute fetch:
/doc-fetcher:fetch-docs tanstack-query --url https://tanstack.com/query/latest/docs - Generate skills:
/doc-fetcher:generate-skill tanstack-query
Example 4: Update Existing Documentation
User: "Update my React documentation"
Your actions:
- Check current cached docs:
/doc-fetcher:list-docs - Update the library:
/doc-fetcher:update-docs react - Regenerate skills if updated:
/doc-fetcher:generate-skill react
Other Useful Commands
List Cached Documentation
/doc-fetcher:list-docs
/doc-fetcher:list-docs --project # Show docs for current project
/doc-fetcher:list-docs --verbose # Show detailed metadata
Update Documentation
/doc-fetcher:update-docs <library> # Update specific library
/doc-fetcher:update-docs --all # Update all cached docs
/doc-fetcher:update-docs --project # Update project dependencies
View/Modify Configuration
/doc-fetcher:config --show # Show current settings
/doc-fetcher:config --set <key> <value>
What NOT to Do
DO NOT Write Custom Crawler Code
All crawling functionality exists in the plugin. Use the slash commands.
DO NOT Use WebFetch for Bulk Crawling
The fetch command handles bulk fetching with rate limiting, robots.txt compliance, and error handling.
DO NOT Skip URL Validation
Always validate URLs before fetching. Use the resolver script or verify with WebSearch.
Summary
- Find URL: Resolver script → Web search → URL patterns
- Fetch docs:
/doc-fetcher:fetch-docs - Generate skills:
/doc-fetcher:generate-skill - Report results: Clear summary with actionable next steps
- Handle errors: Provide solutions, not just error messages
The doc-fetcher plugin has professional-grade infrastructure. Use it via slash commands.
Sponsors
Details
- Marketplace
- squirrelsoft-dev/squirrelsoft-marketplace
- Source
- squirrelsoft-dev/doc-fetcher
- Category
- Documentation