TL;DR

What: GPTBot collects training data; OAI-SearchBot puts your pages into ChatGPT search. They are separate crawlers with separate robots.txt rules.
Why it matters: Blocking the wrong one either removes you from AI answers or doesn’t protect your content at all.
What to do: Allow AI search and assistant bots, decide separately on training bots, then check firewalls and JavaScript rendering.

GPTBot is OpenAI’s crawler for collecting public web content that may be used to train its AI models. OAI-SearchBot is OpenAI’s separate crawler that indexes pages so they can appear in ChatGPT search results. An AI training crawler gathers content for building models, an AI search crawler indexes pages so an assistant can cite them, and a user-triggered agent such as ChatGPT-User fetches a page only when a person asks about it.

The GPTBot vs OAI-SearchBot question trips up a lot of site owners. You read that AI companies train on your content, you add User-agent: GPTBot and Disallow: / to robots.txt, and you feel protected. Then you wonder why ChatGPT never cites your articles. Or the opposite happens: you block every AI bot you can find and quietly disappear from ChatGPT, Claude and Perplexity answers.

The fix is understanding that each AI company now runs several crawlers with very different jobs. OpenAI documents four of them on its crawler documentation page, and Anthropic, Perplexity and Google publish similar lists. In this guide you’ll see what each bot does, which ones to allow, a tested robots.txt template, and two problems robots.txt can’t fix at all. I’ll also share what we found when we ran our AI Agent Readiness Checker against well-known sites this week.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls for model training. OAI-SearchBot crawls for ChatGPT search. Blocking one has no effect on the other.

OpenAI’s documentation describes GPTBot as the crawler used for content “that may be used in training our generative AI foundation models”. OAI-SearchBot is described as the bot used to surface websites in ChatGPT’s search features. Both obey robots.txt, and each reads its own group.

That separation is good news for you. It means you can say “don’t train on my content” without saying “don’t show my content to ChatGPT users”. Most publishers who block AI training still want the traffic and citations that AI search sends.

The takeaway: never treat “AI crawler” as one thing. Decide per job, not per company.

Which AI crawlers exist in 2026, and what does each one do?

The major assistants each run a search crawler, a user-triggered fetcher and a training crawler. Here is the list we track, taken from each operator’s own documentation.

User agent	Operator	Job	Obeys robots.txt?
OAI-SearchBot	OpenAI	ChatGPT search results	Yes
ChatGPT-User	OpenAI	Opens a page when a user asks	May not apply
GPTBot	OpenAI	Model training	Yes
Claude-SearchBot	Anthropic	Claude search quality	Yes
Claude-User	Anthropic	Opens a page when a user asks	Yes
ClaudeBot	Anthropic	Model training	Yes
PerplexityBot	Perplexity	Perplexity search results (not training)	Yes
Perplexity-User	Perplexity	Opens a page when a user asks	Generally no
Google-Extended	Google	Gemini training and grounding only	Yes

Sources: OpenAI, Anthropic, Perplexity and Google. Two details matter most. Perplexity says PerplexityBot is not used to train models. And Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”.

Applebot-Extended (Apple) and CCBot (Common Crawl) are worth adding to a training block list too, since Common Crawl’s archive is widely used to build AI datasets.

Does blocking GPTBot remove your site from ChatGPT search?

No. ChatGPT search depends on OAI-SearchBot, so blocking only GPTBot keeps you eligible for ChatGPT search results.

The same pattern holds at Anthropic. Its support article says that disabling Claude-SearchBot “may reduce your site’s visibility and accuracy in user search results”, while ClaudeBot gathers training data. So a site that blocks ClaudeBot but allows Claude-SearchBot can still be cited in Claude’s answers.

Here is a real example of how easy this is to get wrong. When we ran The Guardian’s homepage through our checker on 29 October 2026, its robots.txt allowed OAI-SearchBot and GPTBot but blocked Claude-SearchBot, Claude-User and PerplexityBot with Disallow: /. That may be a deliberate business decision, but it means those assistants can’t cite the page, while OpenAI’s training crawler is still allowed in. If your goal is “visible in AI answers, not used for training”, that is exactly backwards.

Whatever you choose, make the choice per crawler and write it down.

Which AI crawlers should you block?

For most sites: allow every search and user-triggered AI crawler, and make a separate, deliberate call on training crawlers.

Blogs, documentation and developer tools (like NexGismo): allow everything. Being quoted by assistants is now a real source of visitors, and your content is already public.
Publishers and paid content: allow search and user agents, block training bots, and keep paywalled pages behind authentication.
Stores: allow search and user agents so assistants can recommend your products; training bots are your call.
Private or internal content: don’t rely on robots.txt at all. Use a login.

Remember that blocking user-triggered agents is unreliable. OpenAI says robots.txt “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores” it, because a human asked for the page.

How do you block AI training bots but keep AI search?

Use three groups: one allowing AI search and assistant bots, one blocking training bots, and your normal User-agent: * group for everyone else.

# AI search and assistants: allowed (keeps you citable in ChatGPT, Claude and Perplexity)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# AI model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

We tested this exact file against our RFC 9309 robots.txt parser before publishing. Every AI search and assistant bot can read /blog/ pages but not /wp-admin/. Every training bot is blocked everywhere. Googlebot and Bingbot keep their usual rules. Replace the sitemap URL with your own.

Now the gotcha almost every tutorial misses. Under RFC 9309, a crawler obeys only the most specific group that names it. It does not merge that group with User-agent: *. So this innocent-looking file:

User-agent: *
Disallow: /wp-admin/

# Looks harmless, but GPTBot now ignores the * group entirely
User-agent: GPTBot
Allow: /

lets GPTBot crawl /wp-admin/, because GPTBot now ignores the * group entirely. Our parser confirms it. Whenever you give a bot its own group, copy your * rules into it, as the template above does.

Why can AI crawlers still fail to read your site after you allow them?

Because robots.txt is only a request. Firewalls and JavaScript rendering decide what AI crawlers actually receive.

Firewalls and CDN bot rules. When we checked Allbirds’ homepage on 29 October 2026, its robots.txt allowed every AI crawler. Yet requests identifying as OAI-SearchBot, ClaudeBot and PerplexityBot got HTTP 429 (too many requests), while an ordinary browser request got HTTP 200. Wikipedia answered a ClaudeBot user agent with HTTP 403 while serving browsers normally. Our test sends the bot’s user agent but can’t use the bot’s real IP addresses, so treat this as a strong hint and confirm in your CDN settings or server logs. If you use Cloudflare, our post on Cloudflare’s AI crawler block explains the settings involved.

JavaScript rendering. Vercel’s analysis of AI crawler traffic found that ChatGPT’s and Claude’s crawlers fetch JavaScript files but don’t execute them. GPTBot requested JavaScript in 11.50% of its fetches and ClaudeBot in 23.84%, yet neither ran any of it. If your main content only appears after client-side rendering, those crawlers see an almost empty page. Server-side rendering, static generation or plain HTML fixes it.

Robots.txt says “you may come in”. Your firewall and your rendering decide whether there is anything to read once they do.

How do you check which AI crawlers can reach your site?

Run a page through an AI agent readiness checker, then confirm with Chrome Lighthouse.

Our free AI Agent Readiness Checker evaluates robots.txt rules for 11 AI crawlers using RFC 9309 matching, compares responses to a browser and to AI crawler user agents, checks that content is in the initial HTML, and gives you a copy-paste robots.txt fix that keeps your other restrictions. It also checks llms.txt, structured data and whether agents can use your forms and buttons.

Chrome added an Agentic Browsing category to Lighthouse in May 2026. It runs in a real browser and checks accessible names, layout stability, llms.txt and WebMCP tools, but it doesn’t test robots.txt rules for AI crawlers or firewall blocks. The two complement each other. If you want to go further and let agents use your site’s actions directly, our WebMCP guide walks through it, and our llms.txt generator builds a spec-compliant llms.txt in a few minutes.

GPTBot collects AI training data, while OAI-SearchBot puts pages into ChatGPT search, and each is controlled by its own robots.txt group.
Blocking AI search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot removes a site from those assistants’ answers; blocking training crawlers does not.
User-triggered agents such as ChatGPT-User and Perplexity-User may not obey robots.txt, so private content needs authentication rather than a Disallow rule.
Under RFC 9309 a crawler follows only the most specific group naming it, so a bot-specific group must repeat the rules from the User-agent: * group.
Firewall bot rules and client-side JavaScript rendering can hide a site from AI crawlers even when robots.txt allows them.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot only collects data for training OpenAI's models. ChatGPT search uses a separate crawler, OAI-SearchBot, and OpenAI documents the two as independent robots.txt settings. You can block GPTBot and still appear in ChatGPT search results as long as OAI-SearchBot is allowed.

What is the difference between ClaudeBot and Claude-SearchBot?

ClaudeBot collects public web content that may be used to train Anthropic's models. Claude-SearchBot indexes pages to improve Claude's search answers. Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results, while blocking ClaudeBot only affects training.

Does robots.txt stop ChatGPT-User and Perplexity-User?

Not reliably. Both fetch a page because a person asked the assistant about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. If you must keep a page away from them, put it behind authentication instead.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether your content is used for Gemini training and grounding. AI Overviews are built from the normal Search index, which Googlebot crawls.

Why can AI crawlers still not read my site after I allowed them in robots.txt?

Two common reasons. A firewall or CDN bot rule may block AI crawler user agents before robots.txt is ever read, and AI crawlers such as GPTBot and ClaudeBot don't execute JavaScript, so content rendered only in the browser is invisible to them. Test both with a readiness checker.

Should I put an Allow: / group for GPTBot in robots.txt?

Only if you copy your other rules into it. Under RFC 9309 a crawler obeys only the most specific group that names it, so a GPTBot group containing just Allow: / silently drops every rule in your User-agent: * group, such as Disallow: /wp-admin/, for that bot.

Sources & Official References

The most useful habit here is boring: check your site the way the crawlers see it. A robots.txt that looks right can still be undone by one CDN setting or a JavaScript-only page, and you won’t notice until AI assistants stop citing you. Make your training decision on purpose, keep AI search and assistant bots in, and re-check after every theme, CDN or security-plugin change. Drop a comment below or subscribe to NexGismo for weekly posts like this.