Fetches, caches, and versions documentation from web sources to provide accurate, version-specific context for AI coding agents

version
2.9.1

Install

/plugin install doc-fetcher@squirrelsoft-dev-tools

Run it inside Claude Code after adding the marketplace with /plugin marketplace add squirrelsoft-dev/squirrelsoft-marketplace.

Contents

Discovered from source

Skills 5

Dependency Detector

I automatically detect your project's dependencies and suggest fetching relevant documentation to provide accurate, version-specific AI assistance.

What I Do

I analyze your project to understand what libraries you're using:

  1. Detect Package Managers: Identify npm, yarn, pnpm, bun
  2. Parse Dependencies: Read package.json, requirements.txt, go.mod, etc.
  3. Extract Versions: Get exact versions you're using
  4. Compare with Cache: Check what documentation is already cached
  5. Suggest Fetches: Recommend fetching missing or outdated docs
  6. Smart Prioritization: Focus on major frameworks first

Supported Project Types

JavaScript/TypeScript (Node.js)

Detection files:

  • package.json - npm/yarn/pnpm dependencies
  • package-lock.json - Locked versions
  • yarn.lock - Yarn locked versions
  • pnpm-lock.yaml - pnpm locked versions
  • bun.lockb - Bun locked versions

Parsed fields:

{
  "dependencies": { ... },      // Production deps
  "devDependencies": { ... },   // Dev deps (lower priority)
  "peerDependencies": { ... }   // Peer deps (important for plugins)
}

Prioritized libraries:

  • Frameworks: next, react, vue, svelte, angular, nuxt
  • Meta-frameworks: astro, remix, solid-start
  • Backend: express, fastify, nest, koa
  • Databases: prisma, typeorm, mongoose, drizzle
  • Supabase: @supabase/supabase-js, @supabase/auth-helpers
  • UI: tailwindcss, shadcn-ui, mui, chakra-ui
  • State: redux, zustand, jotai, mobx

Python

Detection files:

  • requirements.txt - pip dependencies
  • pyproject.toml - Poetry/modern Python
  • Pipfile - Pipenv
  • setup.py - Package setup
  • environment.yml - Conda

Prioritized libraries:

  • Frameworks: django, flask, fastapi, tornado
  • Data: pandas, numpy, scipy, matplotlib
  • ML: tensorflow, pytorch, scikit-learn
  • Async: asyncio, aiohttp, celery

Go

Detection files:

  • go.mod - Go modules
  • go.sum - Checksums

Prioritized libraries:

  • Frameworks: gin, echo, fiber, chi
  • gRPC: grpc-go
  • Databases: gorm, sqlx

Rust

Detection files:

  • Cargo.toml - Rust dependencies
  • Cargo.lock - Locked versions

Prioritized libraries:

  • Frameworks: actix-web, rocket, axum
  • Async: tokio, async-std
  • Serde: serde, serde_json

Detection Process

Phase 1: Project Scan

Scanning project for dependencies...

✓ Found package.json
✓ Found package-lock.json (exact versions available)
✓ Detected package manager: npm

Phase 2: Parse Dependencies

// Example package.json
{
  "dependencies": {
    "next": "^15.0.3",
    "react": "^18.2.0",
    "@supabase/supabase-js": "^2.39.0",
    "tailwindcss": "^3.4.0"
  },
  "devDependencies": {
    "typescript": "^5.3.0",
    "@types/react": "^18.2.0"
  }
}

Extracted:

Production dependencies: 4
Development dependencies: 2

Frameworks detected:
  - Next.js (v15.0.3)
  - React (v18.2.0)

Libraries detected:
  - Supabase (v2.39.0)
  - Tailwind CSS (v3.4.0)

Phase 3: Version Resolution

Resolving exact versions...

next: ^15.0.3 → 15.0.3 (from package-lock.json)
react: ^18.2.0 → 18.2.0 (from package-lock.json)
@supabase/supabase-js: ^2.39.0 → 2.39.0 (from package-lock.json)
tailwindcss: ^3.4.0 → 3.4.1 (from package-lock.json)

Phase 4: Cache Comparison

Comparing with cached documentation...

✓ next@15.0.3
  Cached: v15.0.3 ✓ (matches exactly)
  Skill: nextjs-15-expert (active)

✗ react@18.2.0
  Cached: Not cached
  Suggestion: /fetch-docs react 18.2.0

⚠ @supabase/supabase-js@2.39.0
  Cached: v2.38.0 (outdated by 1 minor version)
  Suggestion: /update-docs supabase

✓ tailwindcss@3.4.1
  Cached: v3.4.0 (close enough, minor version)
  Skill: tailwind-expert (active)

Phase 5: Recommendations

Documentation Recommendations:

Priority 1 (Critical frameworks):
  1. /fetch-docs react 18.2.0
     Reason: Core framework, not cached

Priority 2 (Outdated versions):
  2. /update-docs supabase
     Reason: Cached version (2.38.0) is outdated

Priority 3 (Optional):
  3. /fetch-docs typescript 5.3.0
     Reason: Dev dependency, could improve type hints

Quick setup:
  Run all: /fetch-docs react && /update-docs supabase

Or automatically:
  /fetch-docs --project

Smart Prioritization

I prioritize documentation fetching based on:

Priority 1: Critical Frameworks

These are essential and should be fetched immediately:

  • Next.js, React, Vue, Svelte, Angular
  • Django, Flask, FastAPI
  • Express, Fastify, NestJS

Reasoning: These are the foundation of your application. Accurate docs are critical.

Priority 2: Major Libraries

Important but not critical:

  • Supabase, Firebase, AWS SDK
  • Prisma, TypeORM, Drizzle
  • Tailwind CSS, MUI, Chakra UI

Reasoning: Used extensively, but usually have stable APIs.

Priority 3: Utility Libraries

Nice to have:

  • Lodash, Ramda
  • Day.js, Moment
  • Axios, node-fetch

Reasoning: Smaller scope, less likely to have breaking changes.

Priority 4: Dev Dependencies

Lowest priority:

  • TypeScript (mostly stable)
  • Testing libraries
  • Linters and formatters

Reasoning: Less likely to need frequent reference.

Library Mapping

Some libraries have different documentation names:

const libraryMappings = {
  // npm package → documentation name
  "@supabase/supabase-js": "supabase",
  "@supabase/auth-helpers-nextjs": "supabase-auth",
  "next": "nextjs",
  "tailwindcss": "tailwind",
  "@tanstack/react-query": "react-query",
  "@mui/material": "mui",
  "@chakra-ui/react": "chakra-ui"
};

I automatically map these when suggesting fetches.

Version Matching Strategy

How I determine if cached docs are "good enough":

const versionMatch = (cached, required) => {
  const [cMajor, cMinor, cPatch] = cached.split('.');
  const [rMajor, rMinor, rPatch] = required.split('.');

  // Major version MUST match
  if (cMajor !== rMajor) return 'mismatch';

  // Minor version ideally matches
  if (cMinor === rMinor) return 'exact';
  if (Math.abs(cMinor - rMinor) === 1) return 'close';

  return 'outdated';
};

// Results:
// 'exact'    → ✓ Perfect, use cached docs
// 'close'    → ⚠ Close enough, but update recommended
// 'outdated' → ⚠ Should update
// 'mismatch' → ✗ Must fetch correct major version

Auto-Activation

I automatically activate in these scenarios:

1. Project Opens

When you open a project with Claude Code:

Welcome to your Next.js project!

Detected dependencies:
  - Next.js 15.0.3 ✓ (docs cached)
  - React 18.2.0 ✗ (not cached)

Would you like me to fetch React documentation?
  → /fetch-docs react 18.2.0

2. New Dependency Added

When you modify package.json:

Detected new dependency: @tanstack/react-query@5.17.0

This library is not cached yet. Fetch documentation?
  → /fetch-docs react-query 5.17.0

3. /list-docs --project

When you explicitly request project analysis:

/list-docs --project

Project Dependencies Analysis:

From package.json:
  ✓ next@15.0.3 (cached, current)
  ✗ react@18.2.0 (not cached)
  ⚠ @supabase/supabase-js@2.39.0 (cached 2.38.0, outdated)

Recommendations:
  /fetch-docs react 18.2.0
  /update-docs supabase

Integration with Other Skills

I work with other doc-fetcher skills:

User opens project
  ↓
dependency-detector activates
  ↓
Detects: next@15.0.3, react@18.2.0
  ↓
Checks cache:
  - next ✓ cached
  - react ✗ not cached
  ↓
Suggests: /fetch-docs react
  ↓
User runs command
  ↓
llms-txt-finder checks for llms.txt
  ↓
doc-indexer crawls and caches
  ↓
doc-skill-generator creates react-18-expert
  ↓
dependency-detector confirms:
  ✓ All project dependencies documented

Configuration

Configure my behavior in doc-fetcher-config.json:

{
  "dependency_detection": {
    "auto_detect": true,
    "auto_suggest": true,
    "project_types": ["javascript", "typescript", "python", "go", "rust"],
    "include_dev_dependencies": false,
    "prioritize_frameworks": true,
    "version_match_tolerance": "minor",
    "suggest_on_project_open": true,
    "suggest_on_dependency_add": true
  }
}

Output Examples

Clean Project (All Cached)

Project dependency check:

✓ All dependencies have cached documentation

next@15.0.3 → nextjs-15-expert (active)
react@18.2.0 → react-18-expert (active)
@supabase/supabase-js@2.39.0 → supabase-expert (active)

You're all set! Claude has accurate docs for your entire stack.

Missing Docs

Project dependency check:

Missing documentation for 2 dependencies:

✗ react@18.2.0
  Impact: High (core framework)
  Command: /fetch-docs react 18.2.0

✗ @tanstack/react-query@5.17.0
  Impact: Medium (data fetching library)
  Command: /fetch-docs react-query 5.17.0

Quick fix:
  /fetch-docs react 18.2.0 && /fetch-docs react-query 5.17.0

Or batch:
  /fetch-docs --project

Mixed State

Project dependency check:

✓ Cached and current: 3 dependencies
  next@15.0.3, typescript@5.3.0, tailwindcss@3.4.1

⚠ Cached but outdated: 1 dependency
  @supabase/supabase-js@2.39.0
  (cached: 2.38.0, outdated by 1 minor version)
  Command: /update-docs supabase

✗ Not cached: 1 dependency
  react@18.2.0
  Command: /fetch-docs react 18.2.0

Quick fix:
  /update-docs supabase && /fetch-docs react 18.2.0

Use Cases

1. New Project Setup

# Clone a repo
git clone https://github.com/example/app
cd app

# Open with Claude Code
# dependency-detector auto-runs

Detected: next@15.0.3, react@18.2.0, supabase@2.39.0

None cached. Fetch all?
  /fetch-docs --project

# Claude now has accurate context for entire stack

2. Version Upgrade

# Upgrade Next.js
npm install next@latest

# dependency-detector notices change

Dependency version changed:
  next: 15.0.3 → 15.1.0

Documentation needs update:
  /update-docs nextjs

# Claude now knows about new features

3. Adding New Library

# Add React Query
npm install @tanstack/react-query

# dependency-detector notices

New dependency detected: @tanstack/react-query@5.17.0

Fetch documentation?
  /fetch-docs react-query 5.17.0

# Claude can now help with React Query

Troubleshooting

"Dependency detection not working"

  • Check package.json exists in root
  • Verify package.json is valid JSON
  • Check auto_detect enabled in config

"Wrong version detected"

  • Ensure lockfile is up to date (npm install)
  • Check for conflicting version specifiers
  • Manually specify version in /fetch-docs

"Too many suggestions"

  • Disable dev dependencies in config
  • Increase version tolerance
  • Manually select which to fetch

"Library not recognized"

  • Check library mapping in code
  • Use custom URL: /fetch-docs mylib --url https://...
  • Submit issue to add library to mappings

See Also

  • /list-docs --project - View project documentation status
  • /fetch-docs --project - Fetch all missing project docs
  • /update-docs --project - Update all project docs
  • doc-indexer skill - Performs the actual fetching

Documentation Fetcher Skill

You have been activated because the user wants to fetch, update, or manage library documentation.

Your Task

Use the doc-crawler subagent to fetch documentation and generate skills for the requested library.

How to Proceed

Step 1: Parse the User's Request

Extract from the user's message:

  • Library name (e.g., "Next.js", "React", "Axios", "Tailwind")
  • Version (optional - e.g., "15", "19", "latest")
  • Custom URL (optional - if user provides a specific docs URL)

Step 2: Spawn the Doc-Crawler Agent

Use the Task tool to spawn the doc-crawler subagent with a clear prompt:

Fetch the [version|latest] documentation for [library] and generate skills.

Library: [extracted library name]
Version: [extracted version or "latest"]
Custom URL: [if provided, otherwise omit]

After fetching:
1. Generate the expert skill using /doc-fetcher:generate-skill
2. Report the results including number of pages cached and skills generated

Step 3: Report Results

After the agent completes, summarize the results to the user:

  • Number of pages cached
  • Location of cached documentation
  • Skills that were generated
  • Any errors or warnings

Example Prompts That Activate This Skill

User SaysParsed As
"Get the latest documentation for Next.js"library: nextjs, version: latest
"Fetch React 19 docs"library: react, version: 19
"I need the Tailwind CSS documentation"library: tailwindcss, version: latest
"Download documentation for Prisma"library: prisma, version: latest
"Update my Vue.js documentation"library: vue, action: update
"Get docs from https://tanstack.com/query"library: tanstack-query, url: provided

Spawning the Agent

When spawning the doc-crawler agent, use this pattern:

Task tool with subagent_type: "doc-fetcher:doc-crawler"

Prompt: "Fetch the [version] documentation for [library].

Instructions:
1. Find the documentation URL (use resolver script or web search)
2. Run /doc-fetcher:fetch-docs [library] [version] [--url if provided]
3. After fetching, run /doc-fetcher:generate-skill [library] [version]
4. Report:
   - Pages cached
   - Cache location
   - Skills generated
   - Any errors or warnings"

Handling Special Cases

User Provides a URL

If the user provides a URL like "Fetch docs from https://example.com/docs":

  • Extract a library name from the URL or ask the user
  • Pass the URL to the agent with --url flag

Update Request

If the user says "update" instead of "fetch":

  • Instruct the agent to use /doc-fetcher:update-docs instead
  • Regenerate skills after update

Multiple Libraries

If the user requests multiple libraries:

  • Spawn separate agents for each library
  • Summarize all results at the end

Version Not Specified

If no version is specified:

  • Default to "latest"
  • The fetch command will detect the current version

What NOT to Do

  1. DO NOT fetch documentation yourself using WebFetch
  2. DO NOT write custom crawling code
  3. DO NOT skip the doc-crawler agent - it handles all the complexity
  4. DO NOT forget to generate skills after fetching

Success Criteria

The task is complete when:

  1. Documentation is cached to ~/.claude/docs/[library]/[version]/
  2. At least one skill is generated (typically [library]-[version]-expert)
  3. User receives a clear summary of what was done

Documentation Indexer

I orchestrate the crawling, parsing, and indexing of documentation from web sources. I'm the core engine behind the /fetch-docs command.

What I Do

I handle the complete documentation fetching workflow:

  1. Discovery: Find documentation pages via sitemap.xml or navigation
  2. Framework Detection: Identify the documentation framework in use
  3. Crawling: Systematically fetch all documentation pages
  4. Content Extraction: Extract clean content from HTML (remove nav, footer, etc.)
  5. Storage: Organize and save documentation locally
  6. Indexing: Create searchable index and metadata
  7. Validation: Ensure completeness and quality

Workflow

Phase 1: Discovery

First, I determine how to discover all documentation pages:

1. Check for llms.txt (delegate to llms-txt-finder)
   ✗ Not found → Continue

2. Check for sitemap.xml
   ✓ Found at https://example.com/sitemap.xml

3. Parse sitemap
   Found 234 URLs matching /docs/ pattern

4. Filter and validate URLs
   Kept 234 documentation URLs
   Excluded 45 blog/marketing URLs

Phase 2: Framework Detection

I detect the documentation framework to optimize extraction:

Supported Frameworks:

  • Docusaurus - Meta's React-based doc framework

    • Detect: docusaurus.config.js or <meta name="generator" content="Docusaurus">
    • Extract: Main article content, sidebar structure
    • Navigate: Use generated sidebar for page discovery
  • VitePress - Vue-powered static site generator

    • Detect: .vitepress/config.js or <meta name="generator" content="VitePress">
    • Extract: .vp-doc content blocks
    • Navigate: Parse config.js for sidebar structure
  • Nextra - Next.js-based docs framework (used by Next.js itself)

    • Detect: _meta.json files or Next.js patterns
    • Extract: Main content from MDX files
    • Navigate: Follow _meta.json hierarchy
  • GitBook - Popular documentation platform

    • Detect: SUMMARY.md or GitBook meta tags
    • Extract: Main article content
    • Navigate: Parse SUMMARY.md for structure
  • Mintlify - Modern documentation platform (used by Supabase)

    • Detect: Mintlify-specific meta tags
    • Extract: Main content blocks
    • Navigate: API-based navigation discovery
  • ReadTheDocs - Sphinx-based documentation

    • Detect: ReadTheDocs theme or Sphinx meta
    • Extract: .document or .body content
    • Navigate: searchindex.js for page list
  • Custom/Static - Fallback for custom sites

    • Detect: No recognized framework
    • Extract: Heuristic-based content extraction
    • Navigate: Follow internal links carefully

Phase 3: Crawling

Systematic fetching with safeguards:

// Conceptual crawling algorithm
const crawlConfig = {
  maxPages: 500,              // Configurable limit
  delayMs: 1000,              // Rate limiting
  timeout: 30000,             // 30 second timeout per page
  retries: 3,                 // Retry failed pages
  respectRobotsTxt: true,     // Honor robots.txt
  userAgent: "Claude Code Doc Fetcher/1.0"
};

for (const url of discoveredUrls) {
  // Rate limiting
  await delay(crawlConfig.delayMs);

  // Fetch with timeout
  const html = await fetchWithTimeout(url, crawlConfig.timeout);

  // Extract content
  const content = await extractContent(html, framework);

  // Save to cache
  await saveToCache(url, content);

  // Progress update
  updateProgress(currentPage, totalPages);
}

Phase 4: Content Extraction

I extract clean, AI-friendly content:

What I Keep:

  • Main article content
  • Headings (h1-h6) with hierarchy
  • Code blocks with language tags
  • Lists and tables
  • Images (with alt text)
  • Inline code
  • Blockquotes (notes, warnings, tips)

What I Remove:

  • Navigation menus
  • Headers and footers
  • Sidebars
  • Advertisement blocks
  • Cookie notices
  • Social media widgets
  • JavaScript bundles
  • Analytics scripts

Example Extraction:

Input (HTML):

<!DOCTYPE html>
<html>
  <head>...</head>
  <body>
    <nav><!-- Navigation --></nav>
    <main>
      <h1>Server Actions</h1>
      <p>Server Actions are asynchronous functions...</p>
      <pre><code class="language-tsx">
        export async function createUser(formData: FormData) {
          'use server'
          // ...
        }
      </code></pre>
    </main>
    <footer><!-- Footer --></footer>
  </body>
</html>

Output (Markdown):

# Server Actions

Server Actions are asynchronous functions...

```tsx
export async function createUser(formData: FormData) {
  'use server'
  // ...
}

### Phase 5: Storage

I organize documentation in a structured format:

.claude/docs/ └── nextjs/ └── 15.0.3/ ├── index.json # Metadata ├── sitemap.json # Page hierarchy └── pages/ ├── getting-started.md ├── routing/ │ ├── introduction.md │ ├── defining-routes.md │ ├── pages-and-layouts.md │ └── ... ├── data-fetching/ │ ├── fetching.md │ ├── caching.md │ └── ... └── api-reference/ └── ...


**index.json** - Complete metadata:
```json
{
  "library": "nextjs",
  "version": "15.0.3",
  "source_url": "https://nextjs.org/docs",
  "fetched_at": "2025-01-17T10:30:00Z",
  "framework": "nextra",
  "page_count": 234,
  "total_size_bytes": 5242880,
  "llms_txt_url": null,
  "sitemap_url": "https://nextjs.org/sitemap.xml",
  "skill_generated": true,
  "skill_path": "skills/nextjs-15-expert",
  "extraction_strategy": "nextra-optimized",
  "crawl_duration_seconds": 245,
  "errors": 0,
  "warnings": 3
}

sitemap.json - Page hierarchy:

{
  "structure": [
    {
      "title": "Getting Started",
      "path": "getting-started.md",
      "children": []
    },
    {
      "title": "Routing",
      "path": "routing/",
      "children": [
        {
          "title": "Introduction",
          "path": "routing/introduction.md"
        },
        {
          "title": "Defining Routes",
          "path": "routing/defining-routes.md"
        }
      ]
    }
  ]
}

Phase 6: Indexing

Create searchable index for fast lookups:

{
  "index_version": "1.0",
  "created_at": "2025-01-17T10:35:00Z",
  "pages": [
    {
      "path": "routing/defining-routes.md",
      "title": "Defining Routes",
      "headings": [
        "Creating Routes",
        "File Conventions",
        "Dynamic Routes"
      ],
      "keywords": ["routing", "file-system", "app-router", "dynamic"],
      "code_languages": ["tsx", "jsx"],
      "word_count": 1234
    }
  ]
}

Framework-Specific Extraction

Docusaurus

Strategy:
1. Parse docusaurus.config.js for sidebar structure
2. Extract from <article> or .markdown elements
3. Preserve admonitions (:::note, :::tip, etc.)
4. Handle versioned docs (if multiple versions)

Example:
  <article class="markdown">
    <h1>Title</h1>
    <p>Content...</p>
  </article>

  →

  # Title
  Content...

VitePress

Strategy:
1. Parse .vitepress/config.js for navigation
2. Extract from .vp-doc or .content divs
3. Handle frontmatter in markdown source
4. Preserve custom containers

Example selectors:
  - .vp-doc
  - .content
  - main article

Nextra

Strategy:
1. Parse _meta.json files for structure
2. Extract main content from MDX
3. Handle Next.js-specific components
4. Follow nested _meta.json for hierarchy

Example:
  _meta.json:
  {
    "index": "Introduction",
    "routing": "Routing",
    "data-fetching": "Data Fetching"
  }

GitBook

Strategy:
1. Parse SUMMARY.md for table of contents
2. Extract from .page-inner or article elements
3. Handle GitBook-specific markdown extensions
4. Follow chapter structure

Example SUMMARY.md:
  # Summary
  * [Introduction](README.md)
  * [Chapter 1](chapter1.md)
    * [Section 1.1](chapter1/section1.md)

Mintlify

Strategy:
1. Use Mintlify API if available
2. Extract from main content containers
3. Handle MDX components
4. Parse mint.json for navigation

Example selectors:
  - .docs-content
  - main article

ReadTheDocs

Strategy:
1. Parse searchindex.js for page list
2. Extract from .document or .body
3. Handle Sphinx directives
4. Preserve code-block languages

Example selectors:
  - div.document
  - div.body
  - section[role="main"]

Crawling Safeguards

Rate Limiting

Default: 1 second between requests
Configurable: 100ms - 5000ms

Behavior:
- Respect server delays
- Exponential backoff on errors
- Throttle on rate limit responses (429)

Robots.txt Compliance

1. Fetch robots.txt before crawling
2. Parse User-agent rules
3. Respect Disallow directives
4. Honor Crawl-delay if specified

Example robots.txt:
  User-agent: *
  Disallow: /admin
  Crawl-delay: 1

Error Handling

404 Not Found:
  → Skip page, log warning, continue

500 Server Error:
  → Retry with exponential backoff (3 attempts)

Rate Limit (429):
  → Increase delay, retry after specified time

Timeout:
  → Retry with longer timeout

Network Error:
  → Retry, offer to pause/resume

Progress Reporting

Fetching Next.js documentation...

[████████████░░░░░░░░] 156/234 pages (67%)

Current: /docs/app/building-your-application/routing/defining-routes
Speed: 2.5 pages/sec
Elapsed: 1m 2s
Remaining: ~30s
Errors: 0

Content Validation

After crawling, I validate the content:

✓ All pages fetched (234/234)
✓ No missing dependencies
✓ Code blocks properly formatted
✓ Images referenced correctly
✓ Internal links valid
⚠ 3 external links broken (logged)

Quality Score: 98/100

Incremental Updates

When updating existing documentation:

1. Compare current index with new sitemap
2. Identify:
   - New pages (added)
   - Modified pages (changed)
   - Removed pages (deleted)
3. Fetch only changed pages
4. Update index
5. Preserve unchanged content

Example:
  Checking for changes...

  New: 5 pages
  Modified: 12 pages
  Removed: 2 pages
  Unchanged: 215 pages

  Fetching only 17 changed pages...
  (Saves ~90% of time vs full re-crawl)

Configuration

Configure my behavior in doc-fetcher-config.json:

{
  "indexer": {
    "crawl_delay_ms": 1000,
    "max_pages_per_fetch": 500,
    "timeout_ms": 30000,
    "max_retries": 3,
    "respect_robots_txt": true,
    "user_agent": "Claude Code Doc Fetcher/1.0",
    "frameworks": {
      "auto_detect": true,
      "priority": ["docusaurus", "vitepress", "nextra", "gitbook"]
    },
    "extraction": {
      "remove_navigation": true,
      "preserve_code_blocks": true,
      "extract_images": true,
      "follow_redirects": true
    },
    "validation": {
      "check_links": true,
      "validate_code": false,
      "min_content_length": 100
    }
  }
}

Technical Implementation

I leverage these tools:

  • WebFetch: Fetch HTML content from URLs
  • Bash: Run curl commands for headers, robots.txt checking
  • Read/Write: Manage local cache storage
  • Glob: Find and organize cached files
  • Task: Launch doc-crawler agent for complex sites

Example Crawl Session

/fetch-docs nextjs

→ Invoking doc-indexer skill...

[1/7] Discovery
  ✓ Checked for llms.txt (not found)
  ✓ Found sitemap.xml
  ✓ Parsed 234 documentation URLs

[2/7] Framework Detection
  ✓ Detected: Nextra
  ✓ Optimized extraction strategy loaded

[3/7] Robots.txt Check
  ✓ Fetched robots.txt
  ✓ No restrictions for our user-agent
  ✓ Crawl-delay: 1 second (using default)

[4/7] Crawling
  [████████████████████] 234/234 (100%)
  Duration: 4m 12s
  Errors: 0

[5/7] Content Extraction
  ✓ Extracted clean markdown from all pages
  ✓ Preserved 156 code blocks
  ✓ Removed navigation/footer from all pages

[6/7] Storage
  ✓ Saved to .claude/docs/nextjs/15.0.3/
  ✓ Created index.json
  ✓ Created sitemap.json
  Total size: 5.2 MB

[7/7] Validation
  ✓ All pages valid
  ✓ Code blocks properly formatted
  ✓ Links checked (3 external broken - logged)
  Quality: 98/100

✓ Documentation indexed successfully!

Next: Generating skill (auto-enabled)
  Run: /generate-doc-skill nextjs

Troubleshooting

"Framework detection failed"

  • Falls back to generic extraction
  • May be less accurate
  • Consider manual framework hint

"Too many pages (>500)"

  • Increase max_pages_per_fetch in config
  • Or fetch specific sections manually

"Rate limited by server"

  • Automatic: I increase delay automatically
  • Manual: Increase crawl_delay_ms in config

"Content extraction poor quality"

  • Framework detection may be wrong
  • Custom site may need doc-crawler agent
  • Check extraction selectors

See Also

  • llms-txt-finder skill - Checks for AI-optimized docs first
  • doc-crawler agent - Advanced crawling for difficult sites
  • /fetch-docs command - Main entry point
  • /update-docs command - Incremental updates

Documentation Skill Generator

I generate Claude Code skills from cached documentation, creating version-specific expert skills that reference your locally cached docs.

What I Do

I transform cached documentation into functional Claude Code skills:

  1. Analyze Documentation: Read cached docs and extract structure
  2. Generate Skill Content: Create skill description and capabilities
  3. Configure Auto-Activation: Set up smart activation based on project context
  4. Create Metadata: Link skill to cached documentation
  5. Optimize for AI: Structure content for effective AI assistance

Skill Templates

I support multiple templates for different use cases:

1. Expert Template (Default)

Best for: Comprehensive library coverage

Features:
  - Complete topic coverage
  - All API references included
  - Code examples from docs
  - Auto-activation rules
  - Version-specific guidance
  - Migration notes (if available)

Use case:
  "I want Claude to be an expert in Next.js 15"

2. Quick Reference Template

Best for: Fast lookups and common patterns

Features:
  - Condensed documentation
  - Top 20% most-used features
  - Cheat-sheet style
  - Minimal context overhead
  - Fast activation

Use case:
  "I just need quick API references"

3. Migration Guide Template

Best for: Version upgrades

Features:
  - What's new comparison
  - Breaking changes highlighted
  - Before/after examples
  - Deprecation warnings
  - Upgrade checklist

Use case:
  "I'm upgrading from Next.js 14 to 15"

Requirements:
  - Multiple versions cached

4. Troubleshooter Template

Best for: Debugging and problem-solving

Features:
  - Common errors catalog
  - Debug strategies
  - Performance issues
  - Configuration problems
  - Stack trace interpretation

Use case:
  "Help me debug issues faster"

5. Best Practices Template

Best for: Code quality and optimization

Features:
  - Recommended patterns
  - Anti-patterns to avoid
  - Performance optimization
  - Security considerations
  - Accessibility tips

Use case:
  "Ensure I'm using the library correctly"

Skill Generation Process

Step 1: Load Documentation

Reading cached documentation...
  Path: .claude/docs/nextjs/15.0.3/
  Files: 234 markdown files
  Size: 5.2 MB
  Index: index.json loaded
  Sitemap: sitemap.json loaded

Step 2: Analyze Content

// Conceptual analysis
const analysis = {
  topics: extractTopics(pages),           // ["Routing", "Data Fetching", ...]
  apiMethods: extractAPIMethods(pages),   // ["fetch", "cookies", "headers", ...]
  codeExamples: extractCodeBlocks(pages), // 156 code examples
  sections: buildHierarchy(sitemap),      // Nested structure
  keywords: generateKeywords(content),    // ["server actions", "app router", ...]
  version: extractVersion(metadata)       // "15.0.3"
};

Step 3: Generate Skill Content

Based on the selected template, I create the skill file:

Example: Expert Template

---
name: nextjs-15-expert
description: Expert knowledge of Next.js 15.0.3 based on official documentation
version: 15.0.3
library: nextjs
docs_path: .claude/docs/nextjs/15.0.3
auto_activate: true
activation_patterns:
  - "package.json contains next@15"
  - "directory contains app/ or pages/"
  - "imports from 'next'"
---

# Next.js 15 Expert

I have comprehensive, version-specific knowledge of Next.js 15.0.3 based on the complete official documentation.

## Topics I Cover

{/* Auto-generated from documentation structure */}

- **Routing** (App Router & Pages Router)
- **Data Fetching** (Server Components, fetch API, caching)
- **Rendering** (SSG, SSR, CSR, Streaming)
- **Server Actions** (Mutations, form handling)
- **API Routes** (Route handlers, middleware)
- **Optimization** (Images, fonts, scripts)
- **Metadata** (SEO, Open Graph)
- **Configuration** (next.config.js options)

## What I Know

{/* Auto-generated capability list */}

I can help you with:
- Setting up new Next.js 15 projects
- Implementing App Router patterns
- Creating Server Actions for mutations
- Optimizing performance and bundle size
- Configuring metadata for SEO
- Deploying to Vercel or self-hosting
- Migrating from Pages Router to App Router
- Troubleshooting common issues

## Code Examples

{/* Reference to extracted code examples */}

I have access to 156 code examples covering:
- TypeScript and JavaScript
- Server Components
- Client Components
- Server Actions
- Route Handlers
- Middleware
- Configuration

## Version Notes

This skill is specifically for Next.js **15.0.3**.

Key features in this version:
- Partial Prerendering (PPR)
- Enhanced Server Actions
- Improved fetch caching
- Better error handling

I will NOT suggest deprecated patterns from earlier versions.

---

*Generated by doc-fetcher plugin*
*Source: https://nextjs.org/docs*
*Cached: 2025-01-17*

Step 4: Create Metadata

I create a .doc-reference.json file to link the skill to cached docs:

{
  "library": "nextjs",
  "version": "15.0.3",
  "docs_path": ".claude/docs/nextjs/15.0.3",
  "generated_at": "2025-01-17T15:30:00Z",
  "template_used": "expert",
  "source_url": "https://nextjs.org/docs",
  "page_count": 234,
  "code_examples": 156,
  "auto_generated": true,
  "last_regenerated": "2025-01-17T15:30:00Z"
}

Step 5: Configure Auto-Activation

I set up intelligent activation rules:

# When to auto-activate this skill

activation_patterns:
  # package.json dependency
  - "package.json contains next@15"
  - "package.json contains next@^15"

  # Directory structure
  - "directory contains app/"
  - "directory contains pages/"

  # File imports
  - "imports from 'next'"
  - "imports from 'next/server'"
  - "imports from 'next/navigation'"

  # File names
  - "file named next.config.js"
  - "file named middleware.ts"
  - "file matches app/**/*.tsx"

  # User intent
  - "user asks about Next.js 15"
  - "user mentions App Router"
  - "user mentions Server Actions"

activation_priority: high  # high, medium, low
activation_confidence: 0.9  # 0.0 - 1.0

Template Customization

Expert Template

Includes:
  - All topics from documentation
  - Complete API reference
  - All code examples
  - Migration notes
  - Best practices sections

Content structure:
  1. Overview
  2. Topics covered (exhaustive)
  3. What I know (capabilities)
  4. Code examples catalog
  5. Version-specific notes
  6. Usage examples

Activation:
  - Aggressive auto-activation
  - High confidence threshold

Quick Reference Template

Includes:
  - Top 20% most-used features only
  - Essential APIs
  - Common code patterns
  - Quick tips

Content structure:
  1. Brief overview
  2. Core features (condensed)
  3. Common patterns
  4. Quick reference tables

Activation:
  - Manual activation preferred
  - Low overhead

Migration Guide Template

Includes:
  - Version comparison (e.g., 14 → 15)
  - Breaking changes
  - New features
  - Deprecated features
  - Code migration examples

Content structure:
  1. What's new
  2. Breaking changes
  3. Deprecated APIs
  4. Migration checklist
  5. Before/after examples

Activation:
  - On version mismatch detection
  - Manual for upgrade planning

Requirements:
  - Requires multiple versions cached
  - Compares v14 vs v15 docs

Troubleshooter Template

Includes:
  - Common errors from docs
  - Debug strategies
  - Performance issues
  - Configuration gotchas

Content structure:
  1. Error catalog
  2. Debugging tips
  3. Performance checklist
  4. Configuration guide
  5. FAQ

Activation:
  - On error keywords
  - Manual for debugging sessions

Best Practices Template

Includes:
  - Recommended patterns from docs
  - Anti-patterns
  - Performance tips
  - Security guidelines

Content structure:
  1. Do's and Don'ts
  2. Performance optimization
  3. Security best practices
  4. Accessibility considerations
  5. Production checklist

Activation:
  - Manual for code review
  - On "best practice" queries

Content Extraction Intelligence

I intelligently extract different content types:

API Methods

// Detected from documentation
const apiMethods = [
  {
    name: "cookies()",
    signature: "cookies(): RequestCookies",
    description: "Read incoming request cookies",
    availability: "Server Components, Route Handlers",
    example: "const cookieStore = cookies(); const theme = cookieStore.get('theme')"
  },
  // ... more methods
];

Code Examples

// Extracted and categorized
const codeExamples = [
  {
    language: "tsx",
    category: "Server Actions",
    title: "Creating a Server Action",
    code: "export async function createUser(formData: FormData) { ... }",
    sourceUrl: "https://nextjs.org/docs/app/building-your-application/data-fetching/server-actions"
  },
  // ... 155 more examples
];

Topics and Sections

// Built from sitemap hierarchy
const topics = [
  {
    title: "Routing",
    pages: 24,
    subtopics: [
      "Defining Routes",
      "Pages and Layouts",
      "Dynamic Routes",
      "Route Groups",
      "Parallel Routes"
    ]
  },
  // ... more topics
];

Skill Naming

I generate meaningful skill names:

// Naming patterns
const patterns = {
  expert: "{library}-{major-version}-expert",
  quickRef: "{library}-quick-ref",
  migration: "{library}-{old-version}-to-{new-version}",
  troubleshoot: "{library}-troubleshooter",
  bestPractices: "{library}-best-practices"
};

// Examples:
// "nextjs-15-expert"
// "supabase-quick-ref"
// "react-17-to-18"
// "nextjs-troubleshooter"
// "nextjs-best-practices"

Regeneration

When documentation is updated, I intelligently regenerate:

Detected documentation update for Next.js
  Old: 15.0.0
  New: 15.0.3

Regenerating skill: nextjs-15-expert

Changes detected:
  - 12 new pages added
  - 3 pages modified
  - 1 page removed
  - 8 new code examples

Skill updated:
  ✓ New features added to topics
  ✓ Code examples refreshed
  ✓ Version metadata updated
  ✓ Activation rules preserved

Skill ready: nextjs-15-expert

Output Example

Generating skill for Next.js 15.0.3...

[1/5] Loading documentation
  ✓ Read 234 markdown files
  ✓ Loaded metadata and sitemap
  ✓ Parsed code examples

[2/5] Analyzing content
  ✓ Identified 12 major topics
  ✓ Extracted 156 code examples
  ✓ Found 89 API methods
  ✓ Generated 247 keywords

[3/5] Generating skill (expert template)
  ✓ Created skill description
  ✓ Listed all topics and capabilities
  ✓ Included code example references
  ✓ Added version-specific notes

[4/5] Configuring auto-activation
  ✓ Set activation patterns
  ✓ Priority: high
  ✓ Confidence: 0.9

[5/5] Saving skill
  ✓ Written to: skills/nextjs-15-expert/SKILL.md
  ✓ Created metadata: .doc-reference.json
  ✓ Registered with Claude Code

✓ Skill generated successfully!

Skill name: nextjs-15-expert
Location: skills/nextjs-15-expert/
Template: expert
Auto-activation: enabled

The skill is now active. Ask me anything about Next.js 15!

Configuration

Configure my behavior in doc-fetcher-config.json:

{
  "skill_generation": {
    "auto_generate": true,
    "default_template": "expert",
    "auto_activate": true,
    "naming_pattern": "{library}-{major-version}-expert",
    "output_directory": "skills",
    "include_code_examples": true,
    "max_examples_per_topic": 5,
    "generate_metadata": true
  }
}

Advanced Features

Multi-Version Skills

Generate and maintain skills for multiple versions:

/generate-doc-skill nextjs 14.2.0 --output skills/nextjs-14-expert
/generate-doc-skill nextjs 15.0.3 --output skills/nextjs-15-expert

Both skills coexist. Activation rules ensure the correct version is used.

Skill Merging

Combine multiple libraries into one skill:

/generate-doc-skill nextjs,react,typescript --template full-stack

Creates a unified skill with knowledge of all three libraries.

Custom Templates

Create your own templates:

# templates/my-custom-template.yml
name: "{library}-custom"
description: "Custom skill for {library}"
sections:
  - overview
  - quick_start
  - api_reference
activation: manual

Use:

/generate-doc-skill nextjs --template ./templates/my-custom-template.yml

Troubleshooting

"Skill generation failed"

  • Check documentation cache exists
  • Verify index.json is valid
  • Re-fetch docs if corrupted

"Skill too large"

  • Use quick-reference template instead
  • Limit code examples in config
  • Filter to specific topics

"Skill not auto-activating"

  • Check activation patterns in SKILL.md
  • Verify package.json contains dependency
  • Manually activate first time

"Outdated skill content"

  • Run /update-docs <library>
  • Regenerate with /generate-doc-skill <library>

See Also

  • /generate-doc-skill command - Main entry point
  • doc-indexer skill - Creates the cached docs I use
  • /fetch-docs command - Auto-generates skills after fetching

LLMs.txt Finder

I specialize in detecting and fetching AI-optimized documentation formats like llms.txt and claude.txt from documentation websites.

What I Do

I help the doc-fetcher plugin find the most AI-friendly documentation format available:

  1. Check for llms.txt: Look for /llms.txt or /llms-full.txt at the site root
  2. Check for claude.txt: Look for Claude-specific documentation at /claude.txt
  3. Parse and validate: Ensure the file contains valid documentation content
  4. Extract metadata: Pull version info, update dates, and structure from the file
  5. Download content: Fetch the complete content for caching

AI-Optimized Documentation Formats

llms.txt

A standard format designed specifically for AI consumption:

# Library Name

> Version: 1.0.0
> Last Updated: 2025-01-17

## Overview

[Concise description optimized for AI understanding]

## Installation

[Installation instructions]

## Quick Start

[Getting started guide]

## API Reference

[Complete API documentation]

## Examples

[Code examples with context]

Benefits:

  • Pre-formatted for AI parsing
  • Optimized for token efficiency
  • Includes structured metadata
  • Often more concise than web docs

claude.txt

Claude-specific documentation format:

# Library Name for Claude

This documentation is optimized for Claude AI assistant.

[Documentation structured specifically for Claude's capabilities]

Detection Strategy

When given a documentation URL, I check in this order:

  1. https://example.com/llms-full.txt (comprehensive version)
  2. https://example.com/llms.txt (standard version)
  3. https://example.com/claude.txt (Claude-specific)
  4. https://example.com/.well-known/llms.txt (alternative location)
  5. https://example.com/docs/llms.txt (docs subdirectory)

Usage

Automatic Activation

I'm automatically invoked when you run:

/fetch-docs <library>

I run first and check for AI-optimized formats before falling back to web crawling.

Manual Usage

You can also invoke me directly through the skill:

"Check if Next.js has an llms.txt file"
"Find AI-optimized docs for Supabase"
"Does https://react.dev have claude.txt?"

What I Return

If I find an AI-optimized documentation file, I return:

{
  "found": true,
  "type": "llms.txt",
  "url": "https://nextjs.org/llms.txt",
  "size_bytes": 524288,
  "version": "15.0.3",
  "last_updated": "2025-01-15",
  "content_preview": "# Next.js 15.0.3...",
  "should_use": true,
  "reason": "AI-optimized format available, more efficient than crawling"
}

If not found:

{
  "found": false,
  "checked_urls": [
    "https://example.com/llms.txt",
    "https://example.com/claude.txt",
    "https://example.com/.well-known/llms.txt"
  ],
  "fallback": "sitemap.xml",
  "reason": "No AI-optimized format found, falling back to web crawling"
}

Integration with Doc Fetcher

When I find an AI-optimized file:

  1. Skip Crawling: No need to crawl hundreds of pages
  2. Faster Fetching: Single file download vs. many HTTP requests
  3. Better Structure: Pre-formatted for AI understanding
  4. Token Efficient: Content is already optimized for LLM context windows
  5. More Accurate: Maintained specifically for AI assistants

Example Flow

User: /fetch-docs nextjs

Doc Fetcher: Invoking llms-txt-finder skill...

llms-txt-finder:
  ✓ Checking https://nextjs.org/llms.txt
  ✓ Found! (524 KB)
  ✓ Version: 15.0.3
  ✓ Last updated: 2025-01-15

  Recommendation: Use llms.txt instead of crawling
  Benefits:
    - 1 file vs ~234 pages to crawl
    - Pre-optimized for AI
    - Faster download (5 seconds vs 4 minutes)

Doc Fetcher: Using llms.txt...
  ✓ Downloaded and cached
  ✓ Generated skill: nextjs-15-expert

Validation

I validate AI-optimized documentation files for:

  1. Format: Proper markdown structure
  2. Completeness: Contains essential sections (installation, API, examples)
  3. Version Info: Includes version metadata
  4. Size: Reasonable size (not truncated or too small)
  5. Content Quality: Actual documentation, not just a placeholder

If validation fails, I recommend falling back to crawling:

⚠ Found llms.txt but validation failed
  Issue: File is only 2 KB (likely incomplete)
  Recommendation: Fall back to sitemap.xml crawling

Known Sites with AI-Optimized Docs

I maintain awareness of popular libraries that provide AI-optimized documentation:

  • Next.js: /llms.txt ✓
  • Vercel AI SDK: /llms.txt ✓
  • Anthropic Claude: /claude.txt ✓
  • OpenAI: /llms.txt ✓
  • Supabase: Checking...
  • Tailwind CSS: Checking...

This list helps me make intelligent guesses about where to look.

Creating llms.txt for Your Project

If you maintain a library, I can help you create an llms.txt file:

"Help me create an llms.txt file for my library"

I'll generate a template following best practices:

# Your Library Name

> Version: 1.0.0
> Last Updated: 2025-01-17
> Repository: https://github.com/you/your-lib
> Documentation: https://docs.yourlib.com

## Overview

[2-3 sentence description of what your library does]

## Installation

[Installation commands for different package managers]

## Quick Start

[Minimal example to get started]

## Core Concepts

[Key concepts users need to understand]

## API Reference

[Complete API documentation]

## Examples

[Common use cases with code examples]

## Advanced Usage

[Advanced patterns and techniques]

## Troubleshooting

[Common issues and solutions]

## Changelog

[Recent changes and migration notes]

Performance Impact

Using AI-optimized documentation dramatically improves performance:

MetricWeb Crawlingllms.txt
Files fetched100-5001
Time to fetch2-10 min5-30 sec
Network requests100-5001
Size on disk5-50 MB0.5-5 MB
Token efficiencyVariableOptimized

Error Handling

I handle various error scenarios:

  • 404: File doesn't exist, fall back to crawling
  • 403: Access forbidden, try alternative URLs
  • Timeout: Server slow, retry with backoff
  • Invalid format: Not proper documentation, fall back
  • Too large: File >50MB, might be data dump, validate carefully

Configuration

Configure my behavior in doc-fetcher-config.json:

{
  "llms_txt": {
    "enabled": true,
    "check_locations": [
      "/llms-full.txt",
      "/llms.txt",
      "/claude.txt",
      "/.well-known/llms.txt",
      "/docs/llms.txt"
    ],
    "max_size_bytes": 52428800,
    "validation_strict": true,
    "prefer_over_crawling": true
  }
}

Future Enhancements

  • Registry: Maintain a registry of known llms.txt URLs
  • Validation Service: Validate llms.txt files against a schema
  • Generator: Auto-generate llms.txt from existing docs
  • Suggestions: Suggest improvements to existing llms.txt files

Technical Implementation

I use these tools to accomplish my tasks:

  1. WebFetch: Fetch content from URLs
  2. Bash: Check HTTP headers with curl
  3. Read: Read configuration files
  4. Write: Cache fetched llms.txt files

Example Checks

Successful Find

Checking for AI-optimized docs at https://nextjs.org...

✓ https://nextjs.org/llms.txt
  Status: 200 OK
  Size: 524 KB
  Content-Type: text/plain
  Last-Modified: 2025-01-15

✓ Validation passed
  - Valid markdown format
  - Contains version: 15.0.3
  - Comprehensive content
  - Reasonable size

Recommendation: Use llms.txt
Benefits: 40x faster than crawling 234 pages

Not Found

Checking for AI-optimized docs at https://example.com...

✗ https://example.com/llms-full.txt (404)
✗ https://example.com/llms.txt (404)
✗ https://example.com/claude.txt (404)
✗ https://example.com/.well-known/llms.txt (404)

No AI-optimized documentation found.
Falling back to sitemap.xml or web crawling.

See Also

  • doc-indexer skill - Main documentation crawling logic
  • doc-crawler agent - Advanced web crawling for non-standard sites
  • /fetch-docs command - Primary entry point that uses this skill

Commands 5

Doc-Fetcher Configuration

View and modify doc-fetcher plugin configuration settings.

Current Context

  • Current working directory: !pwd

Your Task

When the user invokes this command, follow these steps:

  1. Parse Arguments: The user provided: $ARGUMENTS
    • --show or no arguments: Display current configuration with explanations
    • --set <key> <value>: Update a specific configuration value
    • --reset: Reset configuration to defaults
    • --path: Show the config file path
    • --restore: Restore configuration from a backup (see step 6)

1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.

  1. Run Config Command:

    • Execute: node ~/.claude/plugins/cache/doc-fetcher/scripts/config.js $ARGUMENTS
    • The script handles:
      • Displaying current configuration with descriptions
      • Validating and updating configuration values
      • Resetting to default values
      • Showing config file location
  2. Handle Results:

    • For --show: Display a formatted table of all settings with current values and descriptions
    • For --set: Confirm the change was made and show the new value
    • For --reset: Confirm defaults were restored
    • For --path: Show the full path to the config file
  3. Available Settings:

    SettingTypeDescription
    crawl_delay_msnumberDelay between page fetches (ms). Increase if rate limited.
    max_pages_per_fetchnumberMaximum pages to fetch per library.
    auto_generate_skillsbooleanAutomatically generate skills after fetching docs.
    timeout_msnumberRequest timeout in milliseconds.
    max_retriesnumberNumber of retry attempts on network failure.
    enable_checkpointsbooleanEnable resume for interrupted fetches.
    checkpoint_intervalnumberPages between checkpoint saves.
    respect_robots_txtbooleanHonor robots.txt crawl rules.
    fetch_llms_urlsbooleanFetch individual pages from llms.txt files.
    user_agentstringUser agent string for HTTP requests.
  4. Error Handling:

    • If invalid key: Show list of valid configuration keys
    • If invalid value type: Show expected type and example
    • If config file missing: Will be created with defaults
    • If permission error: Suggest checking file permissions
  5. Handle --restore (DO NOT run the node script for this):

    • List backup files: ls -la ~/.claude/doc-fetcher/
    • If no backups exist, inform the user and exit
    • Display the backups as a numbered list with timestamps (parse from filename format: doc-fetcher-config.YYYY-MM-DDTHH-MM-SS-MMMZ.json)
    • Use AskUserQuestion to ask the user which backup to restore (show numbers 1, 2, 3, etc.)
    • After user selection, copy the selected backup: cp ~/.claude/doc-fetcher/<selected-backup-file> ~/.claude/plugins/cache/doc-fetcher/doc-fetcher-config.json
    • Confirm the restore was successful

Examples

# Show all current settings
/doc-fetcher:config

# Show config with full details
/doc-fetcher:config --show

# Increase crawl delay to avoid rate limiting
/doc-fetcher:config --set crawl_delay_ms 2000

# Disable automatic skill generation
/doc-fetcher:config --set auto_generate_skills false

# Reset all settings to defaults
/doc-fetcher:config --reset

# Show config file location
/doc-fetcher:config --path

# Restore from a previous backup
/doc-fetcher:config --restore

Important Notes

  • Configuration is stored in ~/.claude/plugins/cache/doc-fetcher/doc-fetcher-config.json
  • Automatic backups: Every config change creates a timestamped backup in ~/.claude/doc-fetcher/
  • Use --restore to restore from any previous backup
  • Changes take effect immediately for new commands
  • Boolean values: use true or false
  • Number values: integers only (no quotes needed)
  • String values: can include spaces if quoted

Fetch Documentation

Fetch and cache documentation for a specified library, creating a local snapshot in ~/.claude/docs for accurate, version-specific AI context.

Current Context

  • Current working directory: !pwd

Your Task

When the user invokes this command, follow these steps:

  1. Parse Arguments: The user provided: $ARGUMENTS
    • First argument is the library name (required)
    • Second argument is optional version (e.g., "15.0.0")
    • Optional --url flag for custom documentation URLs
    • If library is missing: prompt for the library name

1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.

  1. Run Fetch Command:

    • Execute: node ~/.claude/plugins/cache/doc-fetcher/scripts/fetch-docs.js $ARGUMENTS
    • The script will:
      • Discover documentation format (llms.txt, claude.txt, or sitemap.xml)
      • Detect the documentation framework (Docusaurus, VitePress, Nextra, etc.)
      • Crawl and cache pages to ~/.claude/docs/[library]/[version]/
      • Respect robots.txt and rate limits
      • Save metadata in index.json and structure in sitemap.json
    • Monitor progress and show status to the user
  2. Handle Results:

    • If successful: Report:
      • Number of pages fetched
      • Storage location (~/.claude/docs/[library]/[version]/)
      • Total size
      • Whether a skill was auto-generated
    • Show next steps (how to use the cached docs or generate a skill)
  3. Error Handling:

    • If URL not found (404): Verify the library name or try --url flag with custom URL
    • If network errors: Suggest retrying or checking connection
    • If rate limited: Suggest increasing crawl_delay_ms in doc-fetcher-config.json
    • If documentation too large (>500 pages): Inform user and ask if they want to continue
    • If framework not detected: Suggest using --url with the docs homepage
    • If script not found: Verify the doc-fetcher plugin is installed
  4. Skill Generation: If auto_generate_skills: true in config, inform the user that a skill was automatically created and how to use it.

Important Notes

  • The script runs from the plugin directory: ~/.claude/plugins/cache/doc-fetcher/
  • Documentation is cached globally in ~/.claude/docs and shared across all projects
  • The cache directory is configured in doc-fetcher-config.json and defaults to ~/.claude/docs
  • Fetching can take several minutes for large documentation sets
  • Use specific versions (e.g., /fetch-docs nextjs 15.0.0) to match your project's package.json
  • The fetched docs are stored locally and won't be updated automatically (use /update-docs for that)

Generate Documentation Skill

Create a Claude Code skill from cached documentation in ~/.claude/docs, enabling version-specific expertise and accurate AI guidance. Skills are saved to ~/.claude/skills.

Current Context

  • Current working directory: !pwd

Your Task

When the user invokes this command, follow these steps:

  1. Parse Arguments: The user provided: $ARGUMENTS
    • First argument is the library name (required)
    • Second argument is optional version (defaults to latest cached version)
    • --template <name>: Skill template to use (default: "expert")
    • --output <path>: Custom output path for the skill
    • If library is missing: prompt for the library name

1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.

  1. Run Generate Skill Command:

    • Execute: node ~/.claude/plugins/cache/doc-fetcher/scripts/generate-skill.js $ARGUMENTS
    • The script will:
      • Locate cached docs in ~/.claude/docs/[library]/[version]/
      • Analyze the documentation structure and content
      • Generate a skill file with version-specific expertise
      • Save skill to ~/.claude/skills/[library]-[version]-[template]/
      • Link the skill to the cached documentation
    • Monitor progress and capture output
  2. Handle Results:

    • If successful: Report:
      • Skill name and location
      • Version the skill references
      • Topics covered
      • Auto-activation rules (if any)
      • How to use the skill
    • Show the generated skill path
  3. Error Handling:

    • If documentation not found: Suggest /fetch-docs [library] to cache docs first
    • If skill already exists: Ask if user wants to overwrite or create a variant
    • If template not found: List available templates and ask user to choose
    • If generation fails: Show error and suggest re-fetching documentation
    • If script not found: Verify the doc-fetcher plugin is installed
  4. Next Steps:

    • IMPORTANT: Tell the user they need to restart Claude Code (exit and reopen) to load the newly generated skills
    • Explain that skills will automatically activate when working with the library once Claude is restarted

Important Notes

  • The script runs from the plugin directory: ~/.claude/plugins/cache/doc-fetcher/
  • Documentation is cached globally in ~/.claude/docs and skills in ~/.claude/skills
  • Skills are available across all projects (global Claude Code skills)
  • The generated skill references the specific version of documentation
  • Skills can auto-activate based on package.json dependencies or file patterns
  • Use --template to customize skill focus (expert, quick-reference, troubleshooter, etc.)
  • Multiple skills can reference the same cached documentation with different templates
  • Regenerate skills after updating documentation to get the latest content

List Documentation

Display all cached documentation from ~/.claude/docs including versions, sizes, update times, and associated skills.

Current Context

  • Current working directory: !pwd

Your Task

When the user invokes this command, follow these steps:

  1. Parse Arguments: The user provided: $ARGUMENTS
    • --project: Show only docs relevant to current project's package.json dependencies
    • --verbose: Show detailed information including page counts, sources, and full metadata
    • If no arguments: show default view (all cached docs)

1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.

  1. Run List Command:

    • Execute: node ~/.claude/plugins/cache/doc-fetcher/scripts/list-docs.js $ARGUMENTS
    • The script will scan ~/.claude/docs and display:
      • Library names and versions
      • Number of pages cached
      • Storage sizes
      • Last update times
      • Associated skills (if any)
    • Capture and display the output
  2. Present Results:

    • Show the formatted output from the list command
    • If using --project flag:
      • Highlight which dependencies are cached and current
      • Identify missing or outdated documentation
      • Suggest fetch/update commands for missing/outdated docs
    • If using --verbose flag:
      • Show detailed metadata (source URLs, frameworks, page counts, local paths)
  3. Provide Suggestions:

    • If docs are outdated: Suggest /update-docs [library]
    • If docs are missing for project dependencies: Suggest /fetch-docs [library]
    • If old versions exist: Mention cleanup options
  4. Error Handling:

    • If cache directory doesn't exist: Inform user no docs are cached yet, suggest fetching docs
    • If cache directory is empty: Suggest using /fetch-docs to start caching documentation
    • If package.json not found (when using --project): Inform user and fall back to showing all cached docs
    • If script not found: Verify the doc-fetcher plugin is installed

Important Notes

  • The script runs from the plugin directory: ~/.claude/plugins/cache/doc-fetcher/
  • Documentation is cached globally in ~/.claude/docs and shared across all projects
  • The list command is fast (reads index files only, doesn't scan page content)
  • The cache directory is configured in doc-fetcher-config.json and defaults to ~/.claude/docs
  • Use --project to see which of your current project's dependencies have cached documentation
  • Use --verbose for complete details including source URLs and local paths

Update Documentation

Check for and update outdated documentation in ~/.claude/docs.

Current Context

  • Current working directory: !pwd

Your Task

When the user invokes this command, follow these steps:

  1. Parse Arguments: The user provided: $ARGUMENTS
    • If no arguments: prompt user to specify a library, use --all, or use --project
    • Valid flags: --all (update all cached docs), --project (update project dependencies), --force (force re-fetch)

1.5. INSTALL DEPENDENCIES: If ~/.claude/plugins/cache/doc-fetcher/node_modules doesn't exist then execute cd ~/.claude/plugins/cache/doc-fetcher && npm install after installation change the directory back to the project workspace root.

  1. Run Update Command:

    • Execute: node ~/.claude/plugins/cache/doc-fetcher/scripts/update-docs.js $ARGUMENTS
    • It will check ~/.claude/docs for cached documentation
    • Wait for completion and capture the output
  2. Handle Results:

    • If successful: Report what was updated (libraries, versions, page counts)
    • If no updates needed: Inform the user their docs are current
    • If errors occur: Show the error and suggest next steps
  3. Error Handling:

    • If documentation not found: Suggest using /fetch-docs first
    • If network errors: Suggest retrying or checking connection
    • If rate limited: Suggest waiting or increasing crawl_delay_ms in config
    • If script not found: Verify the doc-fetcher plugin is installed
  4. Next Steps: If updates were made and skills exist, remind the user that associated skills have been automatically regenerated.

Important Notes

  • The script runs from the plugin directory: ~/.claude/plugins/cache/doc-fetcher/
  • Documentation is cached globally in ~/.claude/docs and shared across all projects
  • The cache directory is configured in doc-fetcher-config.json and defaults to ~/.claude/docs
  • Updates check for newer versions and re-fetch if available
  • Use --force to re-fetch even if recently updated
  • The --project flag reads the current project's dependencies and updates matching docs in the global cache

Agents 1

Documentation Crawler Agent

Your Mission

You are the orchestration agent for the doc-fetcher plugin. Your job is to:

  1. Find the correct documentation URL for the requested library
  2. Fetch documentation using the plugin's slash commands
  3. Generate skills from the cached documentation
  4. Report results to the user

CRITICAL RULES:

  1. Use /doc-fetcher:fetch-docs for fetching documentation
  2. Use /doc-fetcher:generate-skill for generating skills
  3. Use /doc-fetcher:list-docs to check cached documentation
  4. Use /doc-fetcher:update-docs to update existing documentation
  5. NEVER create custom crawler files or write crawling logic
  6. MONITOR command output and report progress to the user
  7. HANDLE errors by adjusting parameters and retrying

Step 1: Finding the Documentation URL

When a user asks you to fetch documentation for a library, you need to find the correct URL. Use these strategies in order:

Strategy A: Check if URL is Provided

If the user provides a URL directly, use it:

User: "Fetch docs from https://nextjs.org/docs"
→ Use: /doc-fetcher:fetch-docs nextjs --url https://nextjs.org/docs

Strategy B: Use the URL Resolver Script

For most libraries, the resolve-docs-url.js script is the best option. It:

  • Queries the SquirrelSoft API (centralized docs URL cache)
  • Checks package registries (npm, PyPI, crates.io)
  • Looks for llms.txt files
  • Validates URLs before returning
node scripts/resolve-docs-url.js <library-name>

Example output:

🔍 Resolving documentation URL for "axios"...
   Ecosystem: npm
   Checking SquirrelSoft registry...
   ✓ Found in SquirrelSoft: https://axios-http.com/docs
   ✓ Resolved: https://axios-http.com/docs (source: squirrelsoft)

Strategy C: Web Search

If the resolver script fails or returns no results, use WebSearch to find the documentation:

Search queries to try:

  1. "[library] official documentation"
  2. "[library] docs site"
  3. "[library] API reference"

What to look for in search results:

  • Official documentation sites (usually [library].dev, [library].io, docs.[library].com)
  • Avoid GitHub README pages (prefer dedicated doc sites)
  • Avoid third-party tutorials or blog posts
  • Look for sites with /docs, /documentation, /api, /guide, /reference paths

Strategy D: Common URL Patterns

If all else fails, try these common patterns:

  • https://[library].dev/docs
  • https://[library].io/docs
  • https://docs.[library].com
  • https://[library].org/docs
  • https://[library]js.org (for JS libraries)

When to Use Each Strategy

ScenarioBest Strategy
User provides URLUse it directly (Strategy A)
Well-known library (react, vue, nextjs)Resolver script (Strategy B)
Library not in registriesWeb search (Strategy C)
Resolver returns GitHub URLTry web search for better docs
New or obscure libraryCombine web search + URL patterns

Step 2: Fetching Documentation

Once you have the documentation URL, use the fetch command:

Basic Fetch (library name only)

/doc-fetcher:fetch-docs <library>

The command will auto-resolve the URL using the resolver script.

Fetch with Specific Version

/doc-fetcher:fetch-docs <library> <version>

Fetch with Custom URL

/doc-fetcher:fetch-docs <library> --url <custom-url>

Fetch with Version and Custom URL

/doc-fetcher:fetch-docs <library> <version> --url <custom-url>

Step 3: Monitoring Output

The fetch command outputs progress in 7 steps:

[1/7] Checking robots.txt...
[2/7] Checking for AI-optimized documentation...
[3/7] Parsing sitemap.xml...
[4/7] Crawling documentation pages...
[5/7] Saving to cache...
[6/7] Saving metadata...
[7/7] Summary

Watch for:

  • ✓ Success messages
  • ⚠ Warnings (skipped pages, robots.txt disallow, etc.)
  • ✗ Errors (404s, network issues, parsing failures)

Step 4: Generating Skills

After documentation is fetched, generate skills:

Generate Default Expert Skill

/doc-fetcher:generate-skill <library>

Generate Specific Version Skill

/doc-fetcher:generate-skill <library> <version>

Generate with Specific Template

/doc-fetcher:generate-skill <library> --template <template-name>

Available Templates:

TemplatePurpose
expertComprehensive coverage (default)
quick-referenceTop 20% most-used features
migration-guideVersion upgrade guidance
troubleshooterCommon errors and debugging
best-practicesRecommended patterns

Step 5: Reporting to User

Summarize the result:

Success:

✓ Successfully fetched [library] v[version] documentation
  - Cached [X] pages to ~/.claude/docs/[library]/[version]/
  - Generated skill: [library]-[version]-expert

Partial Success:

⚠ Fetched [library] documentation with warnings
  - Cached [X] pages ([Y] skipped)
  - Reason: [explain skipped pages]

Failure:

✗ Could not fetch documentation for [library]
  - Error: [explain error]
  - Suggestion: [provide actionable fix]

Step 6: Handling Errors

Error TypeSolution
URL not foundTry web search to find correct URL
No sitemap foundProvide explicit URL with --url flag
Robots.txt blocksInform user - cannot crawl
Network timeoutsSuggest retry - server may be slow
404 errorsURL may be wrong - try alternative
Rate limitingWait and retry (handled automatically)

Example Workflows

Example 1: Standard Documentation Fetch

User: "Get the Next.js 15 documentation"

Your actions:

  1. Run resolver to find URL:
    node scripts/resolve-docs-url.js nextjs
    
  2. Execute fetch command:
    /doc-fetcher:fetch-docs nextjs 15
    
  3. Monitor output and report results
  4. Generate skills:
    /doc-fetcher:generate-skill nextjs 15
    

Report to user: "✓ Successfully fetched Next.js 15 documentation. Cached 234 pages. Generated nextjs-15-expert skill."


Example 2: Unknown Library

User: "Fetch documentation for some-new-lib"

Your actions:

  1. Try resolver script:
    node scripts/resolve-docs-url.js some-new-lib
    
  2. If no result, use WebSearch:
    Search: "some-new-lib official documentation"
    
  3. Find the docs URL from search results
  4. Fetch with explicit URL:
    /doc-fetcher:fetch-docs some-new-lib --url https://some-new-lib.dev/docs
    

Example 3: Custom URL Provided

User: "Fetch docs from https://tanstack.com/query/latest/docs"

Your actions:

  1. Parse library name from URL (tanstack-query)
  2. Execute fetch:
    /doc-fetcher:fetch-docs tanstack-query --url https://tanstack.com/query/latest/docs
    
  3. Generate skills:
    /doc-fetcher:generate-skill tanstack-query
    

Example 4: Update Existing Documentation

User: "Update my React documentation"

Your actions:

  1. Check current cached docs:
    /doc-fetcher:list-docs
    
  2. Update the library:
    /doc-fetcher:update-docs react
    
  3. Regenerate skills if updated:
    /doc-fetcher:generate-skill react
    

Other Useful Commands

List Cached Documentation

/doc-fetcher:list-docs
/doc-fetcher:list-docs --project    # Show docs for current project
/doc-fetcher:list-docs --verbose    # Show detailed metadata

Update Documentation

/doc-fetcher:update-docs <library>  # Update specific library
/doc-fetcher:update-docs --all      # Update all cached docs
/doc-fetcher:update-docs --project  # Update project dependencies

View/Modify Configuration

/doc-fetcher:config --show          # Show current settings
/doc-fetcher:config --set <key> <value>

What NOT to Do

DO NOT Write Custom Crawler Code

All crawling functionality exists in the plugin. Use the slash commands.

DO NOT Use WebFetch for Bulk Crawling

The fetch command handles bulk fetching with rate limiting, robots.txt compliance, and error handling.

DO NOT Skip URL Validation

Always validate URLs before fetching. Use the resolver script or verify with WebSearch.


Summary

  1. Find URL: Resolver script → Web search → URL patterns
  2. Fetch docs: /doc-fetcher:fetch-docs
  3. Generate skills: /doc-fetcher:generate-skill
  4. Report results: Clear summary with actionable next steps
  5. Handle errors: Provide solutions, not just error messages

The doc-fetcher plugin has professional-grade infrastructure. Use it via slash commands.

Sponsors

Details

Marketplace
squirrelsoft-dev/squirrelsoft-marketplace
Source
squirrelsoft-dev/doc-fetcher
Category
Documentation

More from squirrelsoft-dev/squirrelsoft-marketplace

All 4