AI Crawler Check

See which AI crawlers can read your site

One by one, for the crawlers behind ChatGPT, Claude, Perplexity, Copilot and Google: what your robots.txt allows, and whether a firewall turns them away.

Free, no sign-up. Rule-based.

The AI crawlers we check

Answer crawlers

They fetch your pages when someone asks an assistant a question, or build the search index it answers from. Block one and you’re missing from that assistant’s answers.

  • OAI-SearchBot ChatGPT search
  • ChatGPT-User ChatGPT
  • Claude-SearchBot Claude search
  • Claude-User Claude
  • PerplexityBot Perplexity
  • Perplexity-User Perplexity
  • Googlebot Google Search, AI Overviews and Gemini
  • Bingbot Bing and Copilot

Training crawlers

They gather text that future AI models learn from. Blocking them is a fair choice; it means future assistants know less about you.

  • GPTBot ChatGPT (training)
  • ClaudeBot Claude (training)
  • Google-Extended Gemini (training)
  • Applebot-Extended Apple Intelligence (training)
  • CCBot Common Crawl, which many models learn from
  • Meta-ExternalAgent Meta AI (training)

Two ways a site blocks them

robots.txt, a file at the top of your site, tells each crawler by name where it may go. A single line like Disallow: / under the wrong name keeps an assistant out of your whole site.

A firewall can turn a crawler away whatever robots.txt says. Many hosts and CDNs now block AI crawlers by default; Cloudflare has for new sites since July 2025. The check asks for your homepage three times, each request naming one crawler (GPTBot, ClaudeBot, PerplexityBot), to see whether one is turned away.

Let assistants in, keep training out

If you want AI assistants to recommend you but would rather your pages didn’t train future models, a robots.txt like this does both:

# Let AI assistants read the site to answer questions
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Crawlers named here skip your User-agent: * rules, so repeat
# its Disallow lines here, for example:
# Disallow: /wp-admin/

# Keep the site out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

A crawler named in a group follows only that group. So if your robots.txt already keeps everyone out of some pages under User-agent: * (such as Disallow: /wp-admin/), copy those lines into the first group too, or the crawlers named there can read those pages.

Want the whole picture, not just crawlers? The AI Findability Check also reads your business details, sitemap and what your homepage says about you.

Pages built to be found

ConveyLoop writes and publishes pages for your business each month, with the structured data, titles and descriptions that search engines and AI assistants read, and keeps what brings visitors. Plans start at $19/month.