The short answer
AI companies send different bots for different jobs. Training bots (GPTBot, ClaudeBot, CCBot) collect pages to train models. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index AI answers cite and link to. User fetchers (ChatGPT-User, Perplexity-User) open one page because a person asked about it.
To stay in AI answers but out of training, block the training bots in robots.txt and allow the search bots. Blocking GPTBot does not remove you from ChatGPT search.
A bot's name proves nothing: anyone can send "GPTBot". Check the request against the company's own IP list.
- Training. Collects pages to train AI models.
- AI search. Builds the index AI answers cite and link to.
- User fetch. Opens one page because a person asked a question.
- Agent. Browses and clicks for a person, like a browser would.
Source: each company's own crawler page, checked November 4, 2026
Source: each company's own crawler page, checked November 4, 2026Which AI bots should you block?
It depends on what you want. Most sites that sell something want to be cited in AI answers, because a citation can send a visitor. Many of them would rather not have their pages used for training. If that's you, this robots.txt does it:
# Stay in AI search answers, opt out of AI training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Bytespider Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: * Allow: /
Three things to know before you paste it:
- Google and Apple have no separate training bot. Google-Extended and Applebot-Extended are names you use in robots.txt. Googlebot still crawls, and blocking Googlebot itself would take you out of Google Search and AI Overviews.
- User fetchers may not read it. ChatGPT-User, Perplexity-User, Google's user fetchers, Meta-ExternalFetcher and Amzn-User say a request a person asked for may skip robots.txt. To stop those you need a firewall rule.
- Check your CDN too. Cloudflare and other hosts can block AI bots before robots.txt is read. If you want AI search traffic, make sure the search bots are allowed there as well.
The AI crawler list
Every bot on this page, grouped by company. "Prove it's real" is what the company gives you to check a request: an IP list, a reverse DNS name, or a signature.
OpenAI
GPTBot
TrainingCrawls pages that may be used to train OpenAI's models. Blocking it is OpenAI's training opt-out and does not remove you from ChatGPT search.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
OAI-SearchBot
AI searchFinds pages to show and link in ChatGPT search. Block it and your pages stop appearing as sources there (OpenAI says changes take about 24 hours).
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
ChatGPT-User
User fetchOpens a page when someone asks ChatGPT or a custom GPT about it. OpenAI: "robots.txt rules may not apply" because a person started it.
- robots.txt
- May ignore robots.txt
- Prove it's real
- IP list
ChatGPT agent
AgentBrowses for a person with a normal Chrome user agent. It shows who it is only by signing each request for chatgpt.com (Web Bot Auth).
- robots.txt
- Not stated
- Prove it's real
- Signature
Anthropic
ClaudeBot
TrainingCollects pages that may be used to train Claude. Blocking it tells Anthropic to leave your future pages out of training.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
Claude-SearchBot
AI searchIndexes pages for Claude's web search. Blocking it can make you less visible in Claude's answers.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
Claude-User
User fetchOpens a page when someone asks Claude a question that needs it. Anthropic says all its bots honor robots.txt, this one included.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
Googlebot
AI searchGoogle Search's crawler. AI Overviews and AI Mode are built from what it crawls, so blocking it removes you from those too.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list + Reverse DNS
Google-Extended
TrainingNot a separate bot: a name you use in robots.txt. Disallow it and Google keeps your pages out of Gemini training and grounding. Search is not affected.
- robots.txt
- This is the robots.txt name
- Prove it's real
- Nothing to check: no requests of its own
Google user fetchers
User fetchNotebookLM sources, link previews and similar. Google: "these fetchers generally ignore robots.txt rules."
- robots.txt
- May ignore robots.txt
- Prove it's real
- IP list
Google-Agent
AgentAgents on Google's servers that browse and take actions for a person. Google is testing signed requests for agent.bot.goog.
- robots.txt
- May ignore robots.txt
- Prove it's real
- IP list + Signature
Perplexity
PerplexityBot
AI searchFinds and links pages in Perplexity answers. Perplexity says it is not used to train AI models.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
Perplexity-User
User fetchOpens a page when a person asks Perplexity something. Perplexity: it "generally ignores robots.txt rules."
- robots.txt
- May ignore robots.txt
- Prove it's real
- IP list
Microsoft
Bingbot
AI searchBing's search crawler. Microsoft Copilot's web answers are grounded in Bing search, so blocking it can take you out of both.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list + Reverse DNS
Apple
Applebot
AI searchPowers Siri and Spotlight suggestions. Also gathers pages Apple may use for its AI models, unless you disallow Applebot-Extended.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list + Reverse DNS
Applebot-Extended
TrainingNot a separate bot: a robots.txt name. Disallow it and Apple will not use what Applebot crawled to train its models.
- robots.txt
- This is the robots.txt name
- Prove it's real
- Nothing to check: no requests of its own
Meta
Meta-ExternalAgent
TrainingCrawls to train Meta's AI models and to index content for its products. Meta tells you to block it in robots.txt and publishes no IP list.
- robots.txt
- Block in robots.txt
- Prove it's real
- Nothing published
Meta-WebIndexer
AI searchIndexes pages so Meta AI can cite and link them.
- robots.txt
- Block in robots.txt
- Prove it's real
- Nothing published
Meta-ExternalFetcher
User fetchOpens a link a person asked a Meta AI feature about. Meta: it "may bypass robots.txt rules."
- robots.txt
- May ignore robots.txt
- Prove it's real
- Nothing published
Amazon
Amazonbot
TrainingImproves Amazon products. Amazon says what it collects "may be used to train Amazon AI models." It does not support Crawl-delay.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list + Reverse DNS
Amzn-SearchBot
AI searchSearch for Alexa and other Amazon products. Amazon says it is not used to train generative AI.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list
Amzn-User
User fetchFetches a page live to answer a person's Alexa question. Amazon says it may not follow all robots.txt rules.
- robots.txt
- May ignore robots.txt
- Prove it's real
- IP list
ByteDance
Bytespider
TrainingByteDance's crawler, widely reported to collect training data. ByteDance publishes no documentation or IP list we could find, so a robots.txt rule is a request, not a guarantee.
- robots.txt
- Not stated
- Prove it's real
- Nothing published
Common Crawl
CCBot
TrainingBuilds Common Crawl, a free public copy of the web that many AI models are trained on. Blocking it keeps future pages out of that dataset.
- robots.txt
- Follows robots.txt
- Prove it's real
- IP list + Reverse DNS
DuckDuckGo
DuckAssistBot
AI searchReads pages live for DuckDuckGo's AI answers, which cite sources. DuckDuckGo: the data "is not used in any way to train AI models."
- robots.txt
- Block in robots.txt
- Prove it's real
- IP list
Do AI bots follow robots.txt?
The training and search bots from the big companies say they do. The bots that fetch a page because a person asked mostly say they may not, since a person, not a crawler, started the visit.
- Says it follows robots.txt
- Says to block it there, no promise
- Says it may ignore robots.txt
- Says nothing
See the numbers
| Job | Follows | Block there | May ignore | Not stated |
|---|---|---|---|---|
| Training | 6 | 1 | 0 | 1 |
| AI search | 7 | 2 | 0 | 0 |
| User fetch | 1 | 0 | 5 | 0 |
| Agent | 0 | 0 | 1 | 1 |
Source: each company's own crawler page
Meta tells you to block its bots in robots.txt but never says they obey it. ByteDance publishes nothing at all about Bytespider, and site owners widely report it ignoring robots.txt, so if you want it gone, block it at your firewall.
How do you know a bot is real?
The user agent is just text the visitor sends, so a scraper can call itself GPTBot or Googlebot to get past your rules. There are three real checks:
- IP list. The company publishes the addresses its bot uses, and you check the request's address against them. OpenAI, Anthropic, Google, Perplexity, Apple, Amazon, Common Crawl and DuckDuckGo all do this.
- Reverse DNS. Look up the name behind the request's address (it should end in googlebot.com for Googlebot, for example), then look that name up again to confirm it points back to the same address.
- Signature. Newer agents sign each request with a key the company publishes (Web Bot Auth). ChatGPT's agent does this, because its user agent looks like a plain Chrome browser. Google is testing it for Google-Agent.
An IP list is only as good as its last update. Google refreshes its lists every day. Others go months between changes: as of November 10, 2026, Microsoft (Bingbot) was last updated 1,042 days ago, against 7 days for Google (Googlebot).
See the numbers
| List | Address ranges | Last updated |
|---|---|---|
| Google (Googlebot) | 317 | 2026-11-03 |
| Google (user fetchers) | 1,068 | 2026-11-03 |
| Perplexity (PerplexityBot) | 13 | 2026-10-28 |
| OpenAI (ChatGPT-User) | 235 | 2026-10-27 |
| Anthropic (all three bots) | 14 | 2026-10-20 |
| OpenAI (GPTBot) | 18 | 2026-09-22 |
| Apple (Applebot) | 24 | 2026-09-15 |
| DuckDuckGo (DuckAssistBot) | 486 | 2026-09-01 |
| Common Crawl (CCBot) | 5 | 2026-08-11 |
| OpenAI (OAI-SearchBot) | 39 | 2026-01-02 |
| Microsoft (Bingbot) | 28 | 2024-01-03 |
Data: the companies' published lists, read by Harvv
A list that hasn't changed in a while isn't necessarily wrong. But if a real bot starts using new addresses before the list catches up, a strict check will call it fake, so a check should say "couldn't confirm", not "fake", when the list is old.
Which AI bots crawl a real site?
Here's what reached harvv.com, logged on our own server and refreshed every hour. Anthropic's bots made the most requests.
See the numbers
| Company | Requests | Share | Proven real |
|---|---|---|---|
| Anthropic | 554 | 31% | 0 |
| OpenAI | 315 | 18% | 0 |
| Amazon | 233 | 13% | 0 |
| Perplexity | 147 | 8% | 0 |
| ByteDance | 134 | 7% | 0 |
| Apple | 109 | 6% | 0 |
| Common Crawl | 83 | 5% | 0 |
| You.com | 55 | 3% | 0 |
Data: Harvv
Every site sees a different mix, and crawlers come in bursts, so treat this as an example, not a benchmark.
What changes in 2027?
On October 30, 2026, Google said it plans to support a shared robots.txt signal for opting out of AI training, worked out at the IETF, alongside other AI providers in 2027. If you use it and don't mention Google-Extended, Google will treat that as blocking Google-Extended. The company-by-company names on this page keep working, so nothing you set up today needs undoing.
How we keep this page current
- The bot descriptions come from each company's own crawler page, linked on every card, and were checked on November 4, 2026. "Follows robots.txt" means the company says so; "May ignore" means the company says a request may skip it.
- The IP-list ages and address counts are read from the companies' published files. A file's own "last updated" field is the date shown.
- The harvv.com numbers come from our own server log of requests whose user agent names an AI bot. They count requests, not pages, and a bot that hides its name isn't counted. No IP address is stored.
- Spot something wrong or missing? Email hello@harvv.com.