AI Crawler Check
See which AI crawlers can read your site
One by one, for the crawlers behind ChatGPT, Claude, Perplexity, Copilot and Google: what your robots.txt allows, and whether a firewall turns them away.
The AI crawlers we check
Answer crawlers
They fetch your pages when someone asks an assistant a question, or build the search index it answers from. Block one and you’re missing from that assistant’s answers.
OAI-SearchBotChatGPT searchChatGPT-UserChatGPTClaude-SearchBotClaude searchClaude-UserClaudePerplexityBotPerplexityPerplexity-UserPerplexityGooglebotGoogle Search, AI Overviews and GeminiBingbotBing and Copilot
Training crawlers
They gather text that future AI models learn from. Blocking them is a fair choice; it means future assistants know less about you.
GPTBotChatGPT (training)ClaudeBotClaude (training)Google-ExtendedGemini (training)Applebot-ExtendedApple Intelligence (training)CCBotCommon Crawl, which many models learn fromMeta-ExternalAgentMeta AI (training)
Two ways a site blocks them
robots.txt, a file at the top of your site, tells each crawler by name where it may go. A single line like Disallow: / under the wrong name keeps an assistant out of your whole site.
A firewall can turn a crawler away whatever robots.txt says. Many hosts and CDNs now block AI crawlers by default; Cloudflare has for new sites since July 2025. The check asks for your homepage three times, each request naming one crawler (GPTBot, ClaudeBot, PerplexityBot), to see whether one is turned away.
Let assistants in, keep training out
If you want AI assistants to recommend you but would rather your pages didn’t train future models, a robots.txt like this does both:
# Let AI assistants read the site to answer questions
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Crawlers named here skip your User-agent: * rules, so repeat
# its Disallow lines here, for example:
# Disallow: /wp-admin/
# Keep the site out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /A crawler named in a group follows only that group. So if your robots.txt already keeps everyone out of some pages under User-agent: * (such as Disallow: /wp-admin/), copy those lines into the first group too, or the crawlers named there can read those pages.
Want the whole picture, not just crawlers? The AI Findability Check also reads your business details, sitemap and what your homepage says about you.
Pages built to be found
ConveyLoop writes and publishes pages for your business each month, with the structured data, titles and descriptions that search engines and AI assistants read, and keeps what brings visitors. Plans start at $19/month.