Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
AI Crawler & Bot Directory — GeoPromptTracker ※ GeoPromptTracker Tools 19 Guides 41 About The bot directory · 28 crawlers · refreshed monthly Every AI crawler, explained This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker . Respects robo
Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
AI Crawler & Bot Directory — GeoPromptTracker ※ GeoPromptTracker Tools 19 Guides 41 About The bot directory · 28 crawlers · refreshed monthly Every AI crawler, explained This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker . Respects robo
