Summary
Gives Claude the ability to analyze, generate, and check robots.txt files with a focus on AI crawler detection. Exposes five tools: fetch_robots pulls and parses robots.txt from any URL, analyze_robots checks pasted content for bot blocking rules, generate_robots creates new files with custom bot selections, list_ai_bots returns a database of 25+ crawlers including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot, and check_bot_status verifies if a specific bot is blocked on a given site. Useful when you need to audit AI crawler access across domains, quickly generate blocking rules without memorizing user agent strings, or help clients understand which bots are hitting their sites. The bot database is the real value here since these user agents change frequently and most devs don't track them.
Plug Mailtrap into your AI workflow and let it handle the email.
Connect Mailtrap MCP →
Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →
Make coding agent sessions - Searchable, Shareable, Vendor-neutral & Scored.
Try For Free →
Your agent targets a perfect 10 Code Health score. Deterministic. Every commit.
Try For Free →
Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →
belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Features
- Visual Toggle UI — Block or allow AI bots with simple on/off switches
- 20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
- Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
- Generate robots.txt — Create a complete robots.txt with your chosen rules
- MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
- Check Bot Status — Verify if a specific bot is blocked on any website
MCP Tools
| Tool | Description |
|---|---|
fetch_robots | Fetch and analyze a robots.txt from any URL |
analyze_robots | Analyze pasted robots.txt content for AI bot blocking status |
generate_robots | Generate a robots.txt with specified blocked bots and custom rules |
list_ai_bots | List all known AI bots with user-agents, companies, and descriptions |
check_bot_status | Check if a specific bot is blocked on a given website |
Installation
Web App
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run dev
Open http://localhost:3000 in your browser.
MCP Server — Claude Desktop
Add to your claude_desktop_config.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}
MCP Server — Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}
Tracked Bots
The server knows about 25+ bots including:
- AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
- AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
- Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Why robotstxt.ai vs Manual Editing
| Feature | robotstxt.ai | Manual Editing |
|---|---|---|
| Visual toggle UI | Yes | No |
| 20+ AI bots database | Yes | Research yourself |
| Analyze existing robots.txt | Yes | No |
| MCP Server | Yes | No |
| Price | Free | Free but tedious |
More Open Source SEO Tools
| Tool | Description |
|---|---|
| awesome-seo-mcp-servers | Curated list of SEO MCP servers and agent skills |
| indexnow-mcp | Instant URL indexing via IndexNow |
| schema-gen | Schema.org JSON-LD markup generator |
| ai-visibility | AI brand visibility tracker |
License
MIT
Featured
Plug Mailtrap into your AI workflow and let it handle the email.
Connect Mailtrap MCP →Agent, run crypto. Access onchain data & trade routes via 1inch.
Install now →Make coding agent sessions - Searchable, Shareable, Vendor-neutral & Scored.
Try For Free →Your agent targets a perfect 10 Code Health score. Deterministic. Every commit.
Try For Free →Integrate web data into your AI product. One API to scrape website & brand data.
Get API Key Now →belt cli automatically finds the best tools and skills for your agent. image, video, music, tts...
one prompt install →Categories
Registryactive
Packagerobotstxt-ai-mcp
TransportSTDIO
UpdatedMar 21, 2026
