AI Crawlers: How ChatGPT, Claude and Perplexity Read Your Site

AI crawlers are the automated bots that AI companies use to fetch web pages, either to train a model, to build a search index that answers questions, or to retrieve a page live while a user is waiting. GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and Google-Extended are the ones you'll see most. They behave very differently from Googlebot, and treating them as one group is the most common mistake we see.

Most of the advice floating around is written for publishers worried about content theft. If you run marketing at a B2B SaaS company, your problem is the opposite one. You want to be read, quoted and recommended. The question isn't whether to keep AI crawlers out, it's which ones you need in, and whether your site is actually readable when they arrive.

Across B2B SaaS campaigns we run, crawl access is rarely the thing holding a brand back. But when it is broken, nothing else you do matters. A site that ships its product pages through client-side rendering is invisible to every major AI crawler, no matter how good the content is.

What we'll cover

• What are AI crawlers and how do they differ from Googlebot?
• Which AI crawlers are actually hitting your site?
• What's the difference between training, search and user-triggered crawlers?
• Do AI crawlers render JavaScript?
• How do AI crawlers decide which pages to fetch?
• Why doesn't robots.txt control whether you appear in AI answers?
• Should you block AI crawlers?
• What changed with Cloudflare in 2026, and why does 15 September matter?
• How do you verify AI crawler traffic is genuine?
• How do you monitor AI crawler activity on your site?
• Do crawl permissions actually get you mentioned, or just cited?
• Where should a B2B SaaS team start?

What are AI crawlers and how do they differ from Googlebot?

An AI crawler is a bot operated by an AI company that requests pages from your server. The difference from a traditional search crawler is intent. Googlebot fetches your page to rank it and, historically, to send you a click in return. An AI crawler may fetch your page to train a model that never links back to you at all.

That imbalance is now measurable. Cloudflare found that more than half of all internet traffic is non-human, and 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025. The same report notes that mixed-use crawlers, which blend search and training in a single bot, now account for over 36% of crawler activity.

The economics got sharper when Cloudflare published crawl-to-referral ratios. In July 2025, Anthropic crawled roughly 38,000 pages for every visitor it referred back, and Perplexity's ratio moved the wrong way, from 54 crawls per referral in January to 195 by July.

Which AI crawlers are actually hitting your site?

Four vendors do almost all of it. Akamai's analysis of its customer traffic found that four companies account for more than 90% of AI bot requests, with OpenAI alone at 42.4%, more than double the next vendor.

Here's the set worth knowing by name, with what each one is officially for.

One naming trap worth flagging. Google-Extended isn't a bot at all. It's a robots.txt token that controls whether content Googlebot already fetched can be used to improve Gemini. Google states plainly in its special-case crawler documentation that using it doesn't affect your inclusion in Search and isn't a ranking signal.

Volumes have moved fast. Between July 2024 and July 2025, GPTBot's share of all crawler traffic went from 4.7% to 11.7% and Meta-ExternalAgent jumped from 0.9% to 7.5%, while Bytespider collapsed from 14.1% to 2.4%. Any list of bots you wrote 18 months ago is already wrong, which is a good argument for classifying by purpose rather than maintaining a list of names.

What's the difference between training, search and user-triggered crawlers?

This is the distinction that determines everything else, and it's now the industry's official model rather than a blogger's framing. OpenAI, Anthropic and Perplexity all run separate bots for separate jobs, and Cloudflare classifies traffic the same three ways.

Training crawlers

GPTBot, ClaudeBot and Meta-ExternalAgent collect content that may end up in a foundation model. Anthropic's own docs say blocking ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets". Nothing about today's answers changes when you block these.

Search crawlers

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the retrieval indexes that answer engines query at runtime. OpenAI is blunt about the consequence: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User fetch a page because a specific person asked something that needs it. These are the ones with the most direct line to a live buyer, and they're also the ones that behave least like a crawler.

Do AI crawlers render JavaScript?

No. Research by Vercel and MERJ found that none of the major AI crawlers execute JavaScript, covering OpenAI, Anthropic, Meta, ByteDance, Perplexity and Common Crawl. They will happily download your JavaScript files, they just never run them. Google's Gemini is the exception, because it rides on Googlebot infrastructure, and Applebot renders too.

For a B2B SaaS site this is the single highest-impact technical finding in the whole area. If your pricing tiers, integration list, feature comparisons or customer logos are injected client-side, they don't exist as far as ChatGPT or Claude are concerned. Server-side rendering or static generation for anything you want quoted is the fix.

The same study found ChatGPT spent 34.82% of its fetches on 404 pages and Claude 34.16%, against 8.22% for Googlebot. AI crawlers are wasteful. They chase stale URLs and they follow redirects badly, which means a clean sitemap and tidy redirect handling buy you more here than they do in classic SEO.

How do AI crawlers decide which pages to fetch?

Differently depending on the job, and the difference shapes what you should optimise.

Training crawlers go wide. They want coverage, so they work through sitemaps and internal links, favour text-heavy paths like /blog/ and /docs/, and revisit on long cycles. There isn't much you can do to steer them beyond having a clean sitemap and a link structure that doesn't bury pages five clicks deep.

Search crawlers behave more like Googlebot. They build an index, so freshness signals, internal linking and page authority all influence what gets picked up and how often. This is where classic technical SEO for SaaS still pays, because the mechanics haven't changed even though the consumer of the index has.

User-triggered fetchers are the interesting ones. They don't browse, they resolve. A model has already decided your URL is relevant to a live question and is fetching it to read now. That means latency and reliability matter in a way they never did for a training crawl. A page that takes four seconds or returns a soft 404 simply drops out of the answer, and you'll never see it in any report.

One pattern worth designing for: because these fetchers arrive with a specific question in mind, pages that answer a single question cleanly outperform sprawling pillar pages. A dedicated comparison page usually gets pulled into answers more often than the same comparison buried in section nine of a 6,000 word guide.

Why doesn't robots.txt control whether you appear in AI answers?

Because the bots closest to a live buyer say, in writing, that it may not apply to them. OpenAI's crawler documentation states that for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply". Perplexity's crawler docs are equally direct: Perplexity-User "generally ignores robots.txt rules".

That isn't evasion, it's stated policy, and it follows the same logic Google used for user-initiated fetches years ago. The practical consequence is that robots.txt is a training and indexing control, not a visibility control. If you want to keep a page out of a live AI answer, robots.txt is the wrong tool.

Genuine evasion does happen. Cloudflare published research in August 2025 accusing Perplexity of using undeclared crawlers with rotating IPs to bypass no-crawl directives, and delisted PerplexityBot as a verified bot as a result. Perplexity disputed the finding.

Should you block AI crawlers?

For most B2B SaaS companies, no. The maths that makes blocking attractive for a news publisher doesn't apply to you. Publishers monetise pageviews, so an uncompensated crawl is a straight loss. You monetise a pipeline, so a mention in an answer that never sends a click can still produce a demo request a week later.

Blocking is also rarer than the discourse suggests. Ahrefs found GPTBot was the most-blocked AI bot at just 5.89% of sites in its sample.

The publisher side of the argument is worth understanding even if you don't share it. Cloudflare CEO Matthew Prince put it starkly when the company changed its defaults.

Our position across client work is narrower than either camp. Allow every search and user-triggered bot without exception. Treat training bots as a business decision rather than an SEO one, and if you have genuinely proprietary data, benchmark reports, or original research that is your commercial moat, keep that specific set out of training while leaving the rest open.

What changed with Cloudflare in 2026, and why does 15 September matter?

Cloudflare replaced its blunt block-AI-bots switch with a three-way classification of Search, Agent and Training, available on every plan including the free tier. That part is helpful. The part that catches people out is the new defaults.

From 15 September 2026, Training and Agent traffic is blocked by default on pages that display ads for new domains, and crawlers that combine Search with Training will be blocked by any customer who blocks Training. Cloudflare names Googlebot, Applebot and BingBot as caught by this, because all three are mixed-use.

Read that twice. A well-meant decision to stop AI training on your content can knock you out of Google. If your site sits behind Cloudflare, check the Security settings before that date. It takes five minutes and it's the highest-stakes item on this page.

How do you verify AI crawler traffic is genuine?

Spoofing a user agent takes one line of code, so the user-agent string alone proves nothing. Every serious operator publishes IP ranges you can check against.

OpenAI publishes separate JSON files per bot at openai.com/gptbot.json, searchbot.json and chatgpt-user.json. Perplexity publishes ranges and includes step-by-step Cloudflare and AWS WAF instructions in its docs. Anthropic publishes a list too, though it explicitly warns against IP-based blocking because it interferes with reading your robots.txt. Google uses a three-signal method: user agent, source IP, and reverse DNS lookup.

The pattern we use is simple. Match the user-agent token rather than the version number, since OpenAI has already moved GPTBot to 1.4 while most published lists still say 1.1, then confirm the IP against the published range, and refresh those ranges on a schedule rather than hardcoding them.

How do you monitor AI crawler activity on your site?

Server logs remain the ground truth. Filter by user agent, verify by IP, then look at three things: which bots arrive, which URLs they request, and what status codes they get back. Given how much time ChatGPT and Claude spend on 404s, that last one usually produces the first fix.

Beyond logs, the tooling splits into two camps and people mix them up constantly.

Crawl-side tools: who fetched you

Cloudflare's bot analytics and AI Audit give per-bot request counts, verified status and blocking controls without touching a log file, and the basic view is included on the free plan. Vercel and Netlify expose similar bot breakdowns in their analytics. Botify and OnCrawl handle this at enterprise log volumes. If you want one number to watch, use verified search-bot requests per week against your priority URLs.

Answer-side tools: whether you were named

Ahrefs Brand Radar tracks brand mentions and citations across ChatGPT, Perplexity, Gemini and AI Overviews, and sits inside an existing Ahrefs subscription, which makes it the easiest add for teams already paying for Ahrefs. Semrush's AI Visibility Toolkit covers similar ground across ChatGPT, Gemini, Perplexity and AI Overviews and is sold as an add-on to a Semrush plan. Profound is the enterprise option, with agent analytics that tie crawler behaviour to answer appearances, priced accordingly. Peec AI and Otterly are lighter prompt-tracking tools aimed at smaller teams, both starting around the 100 to 200 euro a month mark. Evertune and AthenaHQ focus on brand-level model perception rather than page-level citations.

We go into the trade-offs in our rundown of AI SEO tools and in the practical guide to measuring AI search visibility. Pricing on all of these moves quickly, so treat the figures as a starting point rather than a quote.

Keep the two layers separate in reporting. Crawl data tells you whether you're readable. Visibility data tells you whether you're chosen. Confusing them leads teams to celebrate a GPTBot traffic spike that means nothing.

Do crawl permissions actually get you mentioned, or just cited?

These are two different outcomes and the difference matters more in B2B SaaS than almost anywhere else.

A citation is a link. The model used your page as a source and pointed at it. A mention is the model naming your product inside the answer, with or without a link. When a buyer asks for the best revenue intelligence platform for a 200-person sales team, the mention is the sale. The citation is a footnote.

Crawl access influences citations directly, because a page has to be fetchable to be cited. It influences mentions only indirectly. Models learn which brands to recommend largely from third-party sources you don't control: comparison posts, buyer guides, review sites, community threads. You can have perfect crawl hygiene and still never get named, which is exactly the pattern we see on technically excellent SaaS sites with thin third-party coverage.

Which is why prompt sets and reporting should weight toward bottom-funnel recommendation and comparison queries rather than definitional ones. Ranking for "what is revenue intelligence" is nice. Being named in "best revenue intelligence tools for mid-market" is revenue. We cover that shift in more depth in our guide to brand mentions and in how to rank in AI search results.

Where should a B2B SaaS team start?

Work in this order, because each step is cheap and the first two catch most of the damage.

Start with robots.txt. Confirm no blanket Disallow is catching OAI-SearchBot, Claude-SearchBot, PerplexityBot or any user-triggered fetcher. Then check Cloudflare or your WAF for the same thing, since a network-level block overrides whatever your robots.txt says.

Next, view source on your five highest-value pages, pricing, key feature pages, your main comparison page, with JavaScript disabled. If the content isn't in the HTML, that's your project for the quarter.

Then pull a week of server logs, verify the bots by IP, and fix the 404s they keep hitting. After that, move up a layer: structure pages so answers are liftable, and go build the third-party presence that drives mentions. Our generative engine optimisation guide and the breakdown of content formats that work in AI search pick up from there, and if you want a second pair of eyes on the whole setup, that's what our AI search optimisation service exists for.

Crawl access won't win you anything on its own. It just makes sure the work that does win is visible when it counts.

Frequently asked questions about AI crawlers

How do I stop AI crawlers from training on my content without losing AI search visibility?

Block only the training bots and leave the search and user-triggered ones alone. In robots.txt, disallow GPTBot, ClaudeBot, Meta-ExternalAgent and Google-Extended, while explicitly allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User. Google confirms that Google-Extended has no effect on Search inclusion or rankings, so this split costs you nothing on the traditional side.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects content for training OpenAI's foundation models. ChatGPT's live search results come from a separate crawler, OAI-SearchBot, and pages fetched in the moment by ChatGPT-User. Blocking GPTBot leaves both of those untouched, so your pages can still appear and be cited in ChatGPT answers.

Why is my site being crawled by AI bots but getting no traffic from them?

Because the ratio is genuinely lopsided. Cloudflare measured Anthropic crawling around 38,000 pages for every visitor it referred back in July 2025. Some of that is training crawls, which never send traffic by design. The rest reflects the zero-click reality of AI answers, where the model uses your content to answer without the user ever leaving. Measure mentions and assisted conversions rather than referral sessions.

Do AI crawlers respect robots.txt?

Training and search crawlers from OpenAI, Anthropic and Perplexity all state that they honour robots.txt, and changes typically take about 24 hours to take effect. User-triggered fetchers are the exception, and they say so openly. OpenAI documents that robots.txt rules may not apply to ChatGPT-User, and Perplexity states that Perplexity-User generally ignores robots.txt, because a person requested the fetch rather than a crawler scheduling it.

Will a JavaScript-heavy site show up in ChatGPT or Perplexity?

Only if the content you care about is in the initial HTML response. Vercel and MERJ tested the major AI crawlers and found none of them execute JavaScript. Anything injected client-side, including pricing tables, feature grids and dynamically loaded comparison content, is invisible to them. Server-side rendering or static generation for those pages fixes it.

How can I tell if AI crawler traffic is real or spoofed?

Check the source IP against the operator's published ranges rather than trusting the user-agent string, which anyone can fake. OpenAI publishes separate JSON files for GPTBot, OAI-SearchBot and ChatGPT-User, Perplexity publishes ranges for both of its bots, and Anthropic publishes a combined list. Google adds a reverse DNS check on top. Match the bot token rather than the version number, since versions change without notice.