Crawlforge Mcp Server

31 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.
Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.

⭐ Star us on GitHub to follow along — it genuinely helps others discover the project.

Table of Contents

• Why CrawlForge?
• CrawlForge vs. alternatives
• Quick Start (2 Minutes)
• Available Tools
• Pricing
• Advanced Configuration
• Usage Examples
• Security & Privacy
• Support
• Contributing

🎯 Why CrawlForge?

• 31 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous agent, a unified multi-format scrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.
• Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
• Local-LLM by default — extract_with_llm runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
• LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
• Autonomous agent — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
• Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
• Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

📊 CrawlForge vs. alternatives

Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

🚀 Quick Start (2 Minutes)

1. Install from NPM

npm install -g crawlforge-mcp-server

2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):

crawlforge login

It prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:

npx crawlforge-setup

This will:

• Guide you through getting your free API key
• Configure your credentials securely
• Auto-configure Claude Code and Cursor (if installed)
• Verify your setup is working

Don't have an API key? Get one free at https://www.crawlforge.dev/signup

One-step setup (v4.6.0+): crawlforge init detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use crawlforge init --all --yes to configure every detected client non-interactively.

3. Configure Your IDE (if not auto-configured)

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"]
    }
  }
}

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

export CRAWLFORGE_API_KEY=your_api_key
npm run start:http    # Streamable HTTP endpoint at http://localhost:10000/mcp

Which launch command? npx -y crawlforge-mcp-server needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (npm i -g crawlforge-mcp-server), use the dedicated crawlforge-mcp bin — it resolves on your PATH, so it survives Node/nvm version switches. The bare crawlforge command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run crawlforge mcp to start the server by hand.

📊 Available Tools

CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.

All Tools (API key required)

Ten tools (scrape, fetch_url, extract_content, crawl_deep, batch_scrape, stealth_mode, scrape_with_actions, process_document, deep_research, extract_embedded_state) accept max_inline_chars (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS): a result over it comes back as a preview plus a result_handle for read_result, with the full result kept for 1 hour under ~/.crawlforge/results/ on your own machine — nothing is uploaded.

For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.

↑ Back to top

💳 Pricing

Every tool is metered and requires an API key. New accounts get 1,000 free trial credits — no credit card required to start.

All plans include:

• Access to all 31 tools
• Credits never expire; paid-plan credits roll over month to month
• API access and webhook notifications

View full pricing

🔧 Advanced Configuration

Environment Variables

# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"

# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.

# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434"   # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b"            # optional; unset = pick the best installed model automatically
                                                   # deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
                                                   # conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text"   # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..."                        # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true"                       # skip Ollama entirely and use CSS/keyword fallbacks

# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unknown names/groups are ignored with a warning

# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run

# Optional: your own proxies for the stealth paths that have no caller to ask —
# the scrape escalation stage, the deep_research retry, the agent, browser_session.
# Comma-separated; a proxy passed on a call always wins. CrawlForge supplies none.
export CRAWLFORGE_STEALTH_PROXIES="http://user:pass@gw.provider.net:8080"
# Unrelated to the PROXY_ROTATION_* variables, which belong to the localization tool.

# Optional: turn off the clearance jar (on by default) — see "Stealth engines and proxies"
export CRAWLFORGE_CLEARANCE_JAR="off"

MCP Spec Features

CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:

• Structured output — scrape, map_site, serp_rank, reddit_search, search_web, extract_structured, and crawl_deep return machine-parseable structuredContent alongside the usual text, validated against a published outputSchema; legacy clients keep working off the text.
• Self-correctable errors — invalid tool input now comes back as an isError: true result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
• JSON Schema 2020-12 tool schemas, deterministic tools/list ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
• Icons on the server, its tools, and its prompts.
• Async tasks (experimental) on the four long-running tools — crawl_deep, batch_scrape, deep_research, agent — for clients that support polling; synchronous results are still returned for clients that don't.

See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.

Local-LLM quickstart (extract_with_llm with Ollama)

extract_with_llm defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).

# 1. Install Ollama:  https://ollama.com
# 2. Pull any model from https://ollama.com/library
ollama pull llama3.2

# 3. Discover what's installed (from your MCP client)
#    list_ollama_models()

# 4. Extract — defaults to Ollama with the model from step 2
#    extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })

Stealth engines and proxies

Every stealth entry point — scrape with escalate: true, stealth_mode, browser_session, scrape_with_actions and the deep_research retry — now defaults to auto: use Camoufox (Firefox anti-detect) when its binary is installed, fall back to Chromium when it is not, with the reason in the result's warnings[]. Every result names the engine that actually ran. Naming an engine explicitly behaves as it always has — camoufox fails rather than falling back, and chromium and playwright are the same engine (on scrape's escalate_engine, spell it playwright).

On browser_session and scrape_with_actions the engine applies only with stealth: true — their ordinary path is the standard Chromium pool.

auto is not free. Measured once each on an Apple Silicon Mac on 2026-09-21 (launch plus a context and a blank page): Chromium 148 ms and 253 MB, Camoufox 946 ms and 667 MB — roughly +0.8 s and +400 MB per stealth call. Pin engine: "playwright" for high-volume work on sites that do not block.

CrawlForge supplies no proxies, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with CRAWLFORGE_STEALTH_PROXIES (comma-separated URLs), which the escalation stage, stealth_mode, browser_session, scrape_with_actions, the deep_research retry and the agent tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.

A challenge solved once is not solved again. When a stealth render gets past a Cloudflare or DataDome wall, the vendor's clearance cookies (cf_clearance, __cf_bm, datadome — nothing else) are kept in ~/.crawlforge/stealth-clearance.json (mode 0600) and replayed to the next stealth context with the same engine, User-Agent and proxy, for at most the cookie's own lifetime and never more than 24 hours. A render that still meets the wall drops them. Set CRAWLFORGE_CLEARANCE_JAR=off to disable.

Full detail, measurements and sources: docs/stealth-engines.md.

Stealth extraction for deep_research (Camoufox)

deep_research automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a real fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (RESEARCH_MAX_STEALTH_RETRIES, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.

Engine selection (RESEARCH_STEALTH_ENGINE):

• auto (default) — prefer Camoufox (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
• camoufox — force Camoufox.
• chromium — force the Chromium stealth engine.

Camoufox gets through walls that stop headless Chromium: in a benchmark run on 2026-09-21 from a residential IP it was the only engine to clear Cloudflare Turnstile (indeed.com) and Akamai (harrods.com), both of which blocked Chromium stealth. Neither engine cleared DataDome or an interactive Turnstile, so this is a better fetch path, not a bypass. To enable it, install the optional dependency and run its one-time binary fetch:

# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox

# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetch

Without the Camoufox binary, deep_research silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with RESEARCH_STEALTH_FALLBACK=false.

Note: Hard IP-reputation blocks (e.g. Reddit's edge 403) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide — point the server at your own with CRAWLFORGE_STEALTH_PROXIES. See docs/stealth-engines.md for details.

Manual Configuration

Your configuration is stored at ~/.crawlforge/config.json:

{
  "apiKey": "cf_live_...",
  "userId": "user_...",
  "email": "you@example.com"
}

📖 Usage Examples

Once configured, use these tools in your AI assistant:

"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"

🔒 Security & Privacy

• Secure Authentication: API keys required for all metered tools
• Local Storage: API keys stored securely at ~/.crawlforge/config.json
• HTTPS Only: All connections use encrypted HTTPS
• No Data Retention: We don't store scraped data, only usage logs
• Rate Limiting: Built-in protection against abuse
• Compliance: Respects robots.txt and GDPR requirements

Security & Approvals

• SSRF enforcement: Every scraped URL is validated before the request is sent — http/https only; blocks loopback, RFC1918, IPv6 private/link-local ranges, cloud metadata endpoints (GCP, Azure), and dangerous ports (SSH, SMTP, DNS, MySQL, Postgres, Redis, MongoDB, etc.). Redirects are re-validated each hop, capped at 5.
• Backend endpoint guard (v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list ({crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). Setting CRAWLFORGE_API_URL to an arbitrary host is blocked at parse time.
• Action allowlist: scrape_with_actions accepts only 11 action types (snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select, hover, navigate). No download, file-write, or arbitrary cross-page navigation primitives exist — navigate goes through the same SSRF and robots.txt gate as the initial URL.
• JavaScript gate: The executeJavaScript action throws by default. Set ALLOW_JAVASCRIPT_EXECUTION=true at deploy time to enable (not recommended in production).
• MCP Elicitation (v3.6.0): Four tools request user confirmation before executing expensive operations — deep_research (>50 URLs), batch_scrape (sync mode, >25 URLs), crawl_deep (projected >500 pages), extract_structured (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
• Per-tool credit gating: Every tool is wrapped with withAuth() and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).

See docs/sandboxing-and-approvals.md for the full reference.

Security Updates

v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.

For the full security policy and how to report a vulnerability, see SECURITY.md.

↑ Back to top

🆘 Support

• Documentation: https://www.crawlforge.dev/docs
• Issues: GitHub Issues
• Email: support@crawlforge.dev
• Discord: Join our community

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide first.

Built with ❤️ by the CrawlForge team

Website | Documentation | API Reference