Robotstxt Ai

Gives Claude the ability to analyze, generate, and check robots.txt files with a focus on AI crawler detection. Exposes five tools: fetch_robots pulls and parses robots.txt from any URL, analyze_robots checks pasted content for bot blocking rules, generate_robots creates new files with custom bot selections, list_ai_bots returns a database of 25+ crawlers including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot, and check_bot_status verifies if a specific bot is blocked on a given site. Useful when you need to audit AI crawler access across domains, quickly generate blocking rules without memorizing user agent strings, or help clients understand which bots are hitting their sites. The bot database is the real value here since these user agents change frequently and most devs don't track them.