--- title: "AI Crawler Checker: Can ChatGPT, Claude and Perplexity Read Your Site?" description: "Free AI crawler checker: see if robots.txt lets GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 13 more read your page, plus the blockers robots.txt misses." url: "https://phillipstemann.com/ai-crawler-checker/" language: "en" date_published: "2026-10-15T12:49:46+02:00" date_modified: "2026-10-15T13:45:09+02:00" --- # AI Crawler Checker: Can ChatGPT, Claude and Perplexity Read Your Site? Free tool # Can *ChatGPT, Claude and Perplexity* read your site? Paste a URL and get your AI visibility score in seconds: which AI crawlers your robots.txt lets in, the exact rule behind every answer, and the page-level issues that quietly keep you out of AI answers. Which page should I check? Check AI access Try:nytimes.comwikipedia.orgphillipstemann.com - 17 AI and search crawlers - Follows the robots.txt standard - Free, no sign-up ## What the checker tests - **robots.txt, per crawler.** For each of 17 crawlers it finds the group that applies (its own name or `CODE_OPEN`*`CODE_CLOSE`) and the most specific Allow or Disallow rule for your exact URL. That follows the robots.txt standard (RFC 9309) and Google’s documented rules. - **Whether the page responds**, or blocks automated requests outright. - **noindex and nosnippet** in the HTML or the `CODE_OPEN`X-Robots-Tag`CODE_CLOSE` header. Either one keeps a page out of AI answers even when every crawler is allowed. - **Text visible without JavaScript.** Most AI crawlers read the raw HTML and do not run JavaScript, so content that only appears after scripts run is invisible to them. - **Structured data** (JSON-LD), and whether the site sits behind Cloudflare, whose bot settings can override robots.txt. ## The three kinds of AI crawlers Every AI company runs several bots with different jobs. Blocking one does not block the others, which is why a single “allow AI” or “block AI” switch is rarely what you actually want. | Job | Examples | What blocking it means | | --- | --- | --- | | Search and answers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot | You can drop out of AI search results and citations | | Live fetch for a user | ChatGPT-User, Claude-User, Perplexity-User | The AI cannot open your page when someone asks about it | | Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot | Your content stays out of future training data. AI search is not affected | If you want to be found in ChatGPT, Claude and Perplexity but keep your content out of training, allow the first two rows and block the third. The checker groups its results the same way. ## Why this checker doesn’t pretend to be GPTBot Some tools “test” AI access by requesting your page with GPTBot’s user agent and reporting whether it gets through. That measures the wrong thing. A request from a random server that calls itself GPTBot is a fake GPTBot, and a well-configured firewall should block it. When I checked a week of my own server logs against the IP lists the AI companies publish, **80% of the requests calling themselves GPTBot were fake**, and 100% of those calling themselves Perplexity-User. They were scanners looking for passwords and API keys. If a spoofed request gets through your firewall, that is a security problem, not a sign that AI search can see you. So this checker reports what robots.txt and the page tell real crawlers, and flags firewall layers like Cloudflare that you need to check separately. To see which real AI crawlers visit you, you need your server logs, matched against the published IP lists. ## What to do if an important crawler is blocked 1. **Find the rule.** The checker shows the exact line that blocks each crawler, and whether it comes from that crawler’s own group or from `CODE_OPEN`User-agent: *`CODE_CLOSE`. 2. **Edit robots.txt.** Give the search crawlers their own group with `CODE_OPEN`Allow: /`CODE_CLOSE`, or remove the Disallow line. In WordPress, Rank Math and Yoast both have a robots.txt editor. 3. **Check your CDN or firewall.** On Cloudflare, open Security › Bots and check “Block AI bots” and AI Labyrinth. Firewall rules that match on user agent should use verified-bot categories instead. 4. **Run the check again** after changing anything. Results are cached for 15 minutes. 5. **Confirm in your logs.** Within a few days you should see verified OAI-SearchBot, Claude-SearchBot and PerplexityBot requests for your important pages. ### Does blocking GPTBot keep me out of ChatGPT search? No. GPTBot collects training data. Whether your pages can appear in ChatGPT search is decided by OAI-SearchBot, and live page fetches come from ChatGPT-User. You can block GPTBot and still be cited in ChatGPT. ### What is Google-Extended? Google-Extended is a robots.txt token, not a separate crawler. It controls whether Google may use your content for Gemini models. It does not affect Google Search or your ranking, which are handled by Googlebot. ### Do I need an llms.txt file? It does no harm, but in a week of my own server logs, 764 verified requests from AI crawlers included zero requests for llms.txt. Crawler access, real HTML and clear answers matter far more. ### Why does the checker say a crawler is allowed when my analytics show no AI traffic? Being allowed is only the first step. AI tools also need a reason to fetch and cite your page, and most of their reading never shows up as a visit in analytics. In my logs, ChatGPT fetched my pages 56 times in a week but sent 2 visits. ### Is the checker free? Do you store the URLs I check? It is free and needs no sign-up. Results are cached for 15 minutes so repeat checks are fast; URLs are not stored beyond that and are not used for anything else. Want the full picture for your site, including which AI crawlers really visit and which pages they read? That is part of my [GEO service](/services/geo/).