• Operators Directory – browse 160+ crawlers by company (Google, OpenAI, Anthropic, etc.). See all their bots, safety stats, and official docs in one place.
• Enhanced crawler profiles with "Blocking Impact" analysis and common use cases
• Added 25+ new AI crawlers including latest models and scrapers
• Bulk robots.txt generator – generate allow/block rules for all filtered bots in one click
• RFC 9309 strict compliance – mimics Googlebot's exact "Longest Match" logic
The tool now helps answer "who owns CCBot?" or "show me all OpenAI crawlers" which came up a lot in the original thread.
Still built with Go + Svelte, still free, no signup required.
Feedback welcome!