ClaudeBot is Anthropic's web crawler for collecting training data, and if it's in your server logs, it's doing what it was built for: reading public pages. It's one of several AI crawlers that now visit practice websites, each with a different documented job. This page explains what each one does, how to tell them apart, and the robots.txt choices that make sense for a practice. The interest is real: "claudebot" draws 6,600 searches a month in the US and 880 in Canada, with "gptbot" at 260, "oai-searchbot" at 210, "google-extended" at 140 and "perplexitybot" at 70 (Google Ads Keyword Planner data via DataForSEO, US and Canada, 12-month average to Sep 2026).
In short: AI crawlers are automated visitors run by AI companies. Some collect pages for model training (ClaudeBot, GPTBot), some index pages for AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and some fetch a page because a person asked. robots.txt lets you treat each differently, and blocking a search crawler can remove you from that assistant's answers.
What AI crawlers are and why they visit
A crawler is a program that requests web pages automatically and identifies itself with a user-agent name in each request. AI companies run crawlers for three separate jobs:
- Training: collecting public content that may be used to train future models.
- Search indexing: building an index the assistant searches when it answers a question with live results.
- User-requested fetching: opening a specific page because a person asked the assistant about it.
There's also a fourth kind of name you'll see in robots.txt guides: control tokens, such as Google-Extended and Applebot-Extended. They don't visit your site at all. They let you tell a company how it may use pages its ordinary crawler already collected.
The main crawlers: ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot
This is what each provider's own documentation says, checked October 22, 2026:
| User agent | Company | Documented job | robots.txt |
|---|---|---|---|
| ClaudeBot | Anthropic | Collects content that could contribute to model training | Honored; Crawl-delay supported |
| Claude-SearchBot | Anthropic | Indexes content "to enhance the relevance and accuracy of search responses" | Honored |
| Claude-User | Anthropic | Retrieves pages when a person asks Claude a question | Honored; disabling it stops retrieval for user queries |
| GPTBot | OpenAI | Crawls content "that may be used in training our generative AI foundation models" | Honored |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features | Honored; changes take about 24 hours |
| ChatGPT-User | OpenAI | Certain user actions in ChatGPT and custom GPTs; "not used for crawling the web in an automatic fashion" | "robots.txt rules may not apply" |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search; "not used to crawl content for AI foundation models" | Honored |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question | "Generally ignores robots.txt rules" |
Sources: Anthropic (opens in a new tab), OpenAI (opens in a new tab), Perplexity (opens in a new tab).
A few details matter in practice. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay directive, and warns that blocking its IP addresses "may not work correctly," because it stops the bot reading your robots.txt. OpenAI says that if you allow both OAI-SearchBot and GPTBot, it "may use the results from just one crawl for both use cases." Perplexity says its settings work independently and may take up to 24 hours to take effect.
Google-Extended and Applebot-Extended: opting out of training
Google-Extended is a robots.txt token, not a crawler. Google says it "doesn't have a separate HTTP request user agent string"; Google's regular crawlers do the fetching. Disallowing it controls whether that content may be used to train Gemini models and for grounding, which Google describes as "providing content from the Google Search index to the model at prompt time," in Gemini Apps and the Vertex AI API. Google states that "Google-Extended does not impact a site's inclusion in Google Search" and is not a ranking signal (Google (opens in a new tab)).
That means Google-Extended doesn't remove you from AI Overviews or AI Mode, which are part of Search. For those, Google points to snippet and indexing controls: "nosnippet, data-nosnippet, max-snippet, or noindex" (Google Search Central (opens in a new tab)).
Applebot-Extended works the same way for Apple. Apple says "Applebot-Extended does not crawl webpages"; it only decides how data collected by Applebot may be used to train Apple's foundation models. Pages that disallow it "can still be included in search results" in Spotlight, Siri and Safari, and the rule isn't considered in ranking (Apple Support (opens in a new tab)).
Training crawlers vs search crawlers: why the difference matters
For a practice, these are two separate decisions.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) decide whether an assistant can read and link your pages when it answers a patient's question. OpenAI is explicit: sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Blocking these is the choice most likely to cost you visibility. Our guides to showing up in ChatGPT and Perplexity for business cover what else those answers depend on.
Training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) decide whether your future content may be used to train models. Some owners object to that on principle, and blocking them is a reasonable choice. Anthropic notes that blocking ClaudeBot signals that "the site's future materials should be excluded," so it isn't retroactive.
Neither setting decides whether an assistant recommends you. Assistants also draw on directories, reviews and news, and AI answers vary by phrasing, location and day.
robots.txt rules for a practice website
robots.txt is a plain-text file at the top level of your domain. Anthropic's instructions, for example, say to place the rule "in the top-level directory" of each subdomain you want covered. Three common setups:
1. Allow everything. This is what medfy.ai itself uses; its live robots.txt reads:
User-Agent: *
Allow: /
Sitemap: https://medfy.ai/sitemap.xml
That's an example of the open setting, not a recommendation that fits every practice.
2. Allow search, opt out of training. A middle ground:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
3. Block all AI crawlers. Possible, but it removes your pages from the search features of each assistant you block, while those assistants can still mention you from other sources.
Whichever you choose, remember what robots.txt isn't. Google says it "is not a mechanism for keeping a web page out of Google," and that while "respectable web crawlers obey the instructions," others "might not" (Google Search Central (opens in a new tab)). It's a request, not a lock.
That's why patient information never belongs on public, crawlable pages. Intake forms, portals and anything that identifies a patient must sit behind proper access controls, not behind a robots.txt line. On websites Medfy runs, form submissions are stored in separate, encrypted HIPAA databases in the US or Canada, never on public pages. For the full picture, see our guide to HIPAA-compliant websites.
Spotting AI crawlers in your server logs
Don't expect to find these crawlers in Google Analytics, which is built to count visitors' browsers. The reliable record is your server's access logs, usually available in your hosting control panel or your CDN dashboard.
- Search the user-agent field for the tokens in the table above: ClaudeBot, Claude-SearchBot, Claude-User, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User.
- Read the full string. OpenAI documents GPTBot's as
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and says the version number may change. - Verify the IP address. Anyone can type a famous bot's name into a request. OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic publishes a list at claude.com/crawling/bots.json; Anthropic says an IP on that list "indicates that the crawler is coming from Anthropic."
- Don't look for Google-Extended or Applebot-Extended. Neither appears in logs, because neither makes its own requests.
If you'd rather not read logs, the free marketing audit checks whether the crawlers behind ChatGPT, Perplexity, Claude and Gemini can read your site. A related file, llms.txt, is often mentioned alongside robots.txt; it doesn't control crawler access. For tools that track what assistants actually say, see our overview of AI visibility tools.
Frequently asked questions
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for training data. Anthropic says it collects web content that could contribute to training its generative AI models, and that blocking it in robots.txt signals your future content should be excluded. It is separate from Claude-SearchBot, which indexes content for search answers, and Claude-User, which fetches pages when a person asks Claude a question.
Should I block GPTBot on my practice website?
It's a policy choice, not a visibility fix. OpenAI says GPTBot crawls content that may be used to train its foundation models, and blocking it opts your site out of that. It doesn't affect OAI-SearchBot, which OpenAI uses to surface sites in ChatGPT search. A practice that objects to training can block GPTBot and still allow OAI-SearchBot. Public pages should never hold patient information either way.
Does blocking AI crawlers stop ChatGPT from mentioning my practice?
Not entirely. OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT can still name you from other sources, such as directories, reviews or news coverage, or from what a model learned earlier. Blocking GPTBot alone affects training, not search answers. Neither choice decides whether an assistant recommends you.
What is Google-Extended?
Google-Extended is a robots.txt control token, not a separate crawler. Disallowing it tells Google not to use content it crawls for training Gemini models or for grounding answers in Gemini Apps and the Vertex AI API. Google says it doesn't affect inclusion in Google Search and isn't a ranking signal. To limit what AI Overviews show, Google points to nosnippet, max-snippet or noindex.