The short answer
Each AI company runs several crawlers, and they do different jobs, so one robots.txt line can cut you out of AI search or only out of model training depending on which name you block. OpenAI, Perplexity, Anthropic, and Google each publish their own list. The table below puts the ten that matter to a home-service website side by side, with a link to the vendor page for each row.
The ten crawlers side by side
The "what blocking it does" column is summarized from each vendor's documentation.
| Crawler | Operator | What it does | What blocking it does | Source |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search results | Site is not shown in ChatGPT search answers | OpenAI bots page |
| GPTBot | OpenAI | Gathers content that may be used to train foundation models | Signals your content should not be used for training; search is unaffected | OpenAI crawler list |
| ChatGPT-User | OpenAI | Visits a page when a person asks ChatGPT a question | OpenAI says it is not used to decide Search appearance; robots.txt rules may not apply because a person triggers the visit | OpenAI user agent notes |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results | Removes the route Perplexity built for linking your site in search | Perplexity bots guide |
| Perplexity-User | Perplexity | Supports a user action inside Perplexity | Generally ignored, because a person requested the fetch | Perplexity user fetcher |
| Claude-SearchBot | Anthropic | Crawls the web to improve search result quality | Content is not indexed for search optimization | Anthropic crawler article |
| Claude-User | Anthropic | Supports Claude users who ask about the web | Claude cannot retrieve your content for that user's query | Anthropic user agent |
| ClaudeBot | Anthropic | Collects material for model training | Future material is excluded from training datasets | Anthropic training crawler |
| Googlebot | Crawls for Google Search | Affects Google Search, including AI features that rely on the Search index | Google crawler reference | |
| Google-Extended | Product token for managing use in Gemini training and grounding | No effect on inclusion in Google Search or on ranking | Google token reference |
What the vendors say in their own words
OpenAI separates search from training. "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers," and "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Each setting is independent, so a contractor can allow OAI-SearchBot and disallow GPTBot at once. For ChatGPT-User, the page says "ChatGPT-User is not used to determine whether content may appear in Search."
Perplexity describes PerplexityBot as "designed to surface and link websites in search results on Perplexity" and adds that it "is not used to crawl content for AI foundation models." Its separate Perplexity-User fetcher "generally ignores robots.txt rules" because a person asked for the fetch.
Anthropic's article describes three agents. Of ClaudeBot it says that disallowing it "signals that the site's future materials should be excluded from our AI model training datasets." Of Claude-User, "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query." Of Claude-SearchBot, disabling it "prevents our system from indexing your content for search optimization."
Google keeps the line clearest. Crawling preferences for Googlebot "affect Google Search," and Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Google's AI features documentation says "a page must be indexed and eligible to be shown in Google Search with a snippet" to appear in AI Overviews and AI Mode, so Googlebot is the one to leave open. More on that in how Google AI Overviews pick local businesses.
A sensible default for a contractor website
The decision comes down to two questions: do you want to be shown in AI search, and do you object to training use? Most owner-led contractors want the first. A default that follows the vendor documentation allows OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Googlebot, and treats GPTBot, ClaudeBot, and Google-Extended as a separate policy decision about training. Blocking the training agents is a legitimate choice, and it leaves ChatGPT search (OAI-SearchBot), Perplexity search (PerplexityBot), and Google Search (Googlebot) unaffected. One caution: Google-Extended controls use of content for Gemini training and grounding, so blocking it can affect Gemini grounding, though not Google Search inclusion or ranking.
Two cautions apply. First, a robots.txt file is a request, and the user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow it, so it is not a privacy control. Second, many hosts, security plugins, and content delivery networks add bot-blocking rules without telling the owner, so a site can be closed to every AI agent by accident. Our free audit reads the robots.txt rules for AI crawlers on a submitted site, which is the quickest way to find out what yours says today. Letting a bot in makes a site eligible without deciding whether a business is named, and none of these vendors publishes a local ranking formula (see does ChatGPT recommend local businesses and the 2026 AI search playbook for home services). Making pages worth citing is the job of answer-first content and FAQ libraries; everything else is in the resources library.
Questions about AI crawlers
Should a contractor block GPTBot?
- It is a policy choice, not a visibility fix. OpenAI says disallowing GPTBot means content should not be used to train its foundation models, and that it is controlled separately from OAI-SearchBot. If you object to training use, block it. If you want ChatGPT search to show your site, leave OAI-SearchBot allowed.
Does blocking Google-Extended hurt my Google rankings?
- No. Google states Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal there. It manages whether content is used for training certain Gemini models and for grounding. Blocking Googlebot is a different matter and would affect Search, so keep the two separate.
How do I see which AI crawlers my site blocks?
- Open your robots.txt file at yourdomain.com/robots.txt and look for the agent names in the table. Then check your host, security plugin, and content delivery network settings, since they can block bots without appearing in that file. The free audit scan reads the robots.txt rules for AI crawlers automatically.