What the checker tests

• robots.txt, per crawler. For each of 17 crawlers it finds the group that applies (its own name or *) and the most specific Allow or Disallow rule for your exact URL. That follows the robots.txt standard (RFC 9309) and Google’s documented rules.
• Whether the page responds, or blocks automated requests outright.
• noindex and nosnippet in the HTML or the X-Robots-Tag header. Either one keeps a page out of AI answers even when every crawler is allowed.
• Text visible without JavaScript. Most AI crawlers read the raw HTML and do not run JavaScript, so content that only appears after scripts run is invisible to them.
• Structured data (JSON-LD), and whether the site sits behind Cloudflare, whose bot settings can override robots.txt.

The three kinds of AI crawlers

Every AI company runs several bots with different jobs. Blocking one does not block the others, which is why a single “allow AI” or “block AI” switch is rarely what you actually want.

Job	Examples	What blocking it means
Search and answers	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot	You can drop out of AI search results and citations
Live fetch for a user	ChatGPT-User, Claude-User, Perplexity-User	The AI cannot open your page when someone asks about it
Model training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot	Your content stays out of future training data. AI search is not affected

If you want to be found in ChatGPT, Claude and Perplexity but keep your content out of training, allow the first two rows and block the third. The checker groups its results the same way.

Why this checker doesn’t pretend to be GPTBot

Some tools “test” AI access by requesting your page with GPTBot’s user agent and reporting whether it gets through. That measures the wrong thing. A request from a random server that calls itself GPTBot is a fake GPTBot, and a well-configured firewall should block it.

When I checked a week of my own server logs against the IP lists the AI companies publish, 80% of the requests calling themselves GPTBot were fake, and 100% of those calling themselves Perplexity-User. They were scanners looking for passwords and API keys. If a spoofed request gets through your firewall, that is a security problem, not a sign that AI search can see you.

So this checker reports what robots.txt and the page tell real crawlers, and flags firewall layers like Cloudflare that you need to check separately. To see which real AI crawlers visit you, you need your server logs, matched against the published IP lists.

What to do if an important crawler is blocked

1. Find the rule. The checker shows the exact line that blocks each crawler, and whether it comes from that crawler’s own group or from User-agent: *.
2. Edit robots.txt. Give the search crawlers their own group with Allow: /, or remove the Disallow line. In WordPress, Rank Math and Yoast both have a robots.txt editor.
3. Check your CDN or firewall. On Cloudflare, open Security › Bots and check “Block AI bots” and AI Labyrinth. Firewall rules that match on user agent should use verified-bot categories instead.
4. Run the check again after changing anything. Results are cached for 15 minutes.
5. Confirm in your logs. Within a few days you should see verified OAI-SearchBot, Claude-SearchBot and PerplexityBot requests for your important pages.

Want the full picture for your site, including which AI crawlers really visit and which pages they read? That is part of my GEO service.