Search & Web Crawling MCP Servers
MCP servers that give your AI agent access to search engines and web crawling tools. From API-based search with Exa and Tavily to full-site crawling with Crawl4AI, these servers let Claude find and extract information from the web.
FAQ
How do search MCP servers differ from browser automation?
Search MCP servers provide structured search results through APIs — they're faster and more reliable for finding information. Browser automation servers control a full browser for interactive tasks. Use search servers when you need to find information; use browser servers when you need to interact with a specific website.
Which search providers are available?
You'll find servers for Exa, Tavily, DuckDuckGo, Brave Search, SearXNG, and Google Search API, among others. Crawling tools like Crawl4AI, Firecrawl, and Jina are also available for extracting content from specific URLs.
Can Claude crawl entire websites?
Yes — crawling MCP servers like Crawl4AI and Firecrawl can traverse websites, follow links, and extract structured content. They're useful for building datasets, monitoring changes, and indexing content.
