Your website can block ChatGPT without you ever choosing to. If your site’s robots.txt file tells OpenAI’s search crawler, OAI-SearchBot, to stay out, or a security service such as Cloudflare blocks AI crawlers for you, ChatGPT can’t read your pages and has less to go on when someone asks for a contractor in your area. Checking takes a browser and, if you use Cloudflare, a look at one setting.
The fix is usually a small change to one file or one setting. Just as important is knowing which bots to let in: the AI companies run separate crawlers for answering questions and for training their models, and you can treat them differently.
Search crawlers and training crawlers are different bots
A crawler is a program that visits web pages and reads them. Each AI company runs more than one, and each identifies itself by a name your website can allow or block.
| Company | Crawler that puts you in answers | Crawler that collects training data |
|---|---|---|
| OpenAI (ChatGPT) | OAI-SearchBot |
GPTBot |
| Anthropic (Claude) | Claude-SearchBot |
ClaudeBot |
| Perplexity | PerplexityBot |
none; Perplexity says PerplexityBot isn’t used for training |
| Google (Search, AI Overviews, AI Mode) | Googlebot |
Google-Extended, a robots.txt setting rather than a separate crawler (also covers Gemini’s answers, see below) |
The one that matters most for being recommended is the search crawler. OpenAI says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers”, while blocking GPTBot keeps your pages out of training, and the two settings are independent (OpenAI). Anthropic says the same split applies to Claude: blocking Claude-SearchBot stops it indexing your content for search, and blocking ClaudeBot keeps it out of training (Anthropic).
Google is the exception. Its Google-Extended setting controls whether your content may be used to train Gemini models and to ground the Gemini app’s answers in what it finds. Google says it doesn’t affect inclusion or ranking in Google Search (Google).
Step 1: read your robots.txt
robots.txt is a small text file at the root of every website that tells crawlers what they may visit. To read yours:
- Type your web address followed by
/robots.txtinto a browser, for exampleyourcompany.com/robots.txt. - Look for any line that starts with
User-agent:and names one of the crawlers above, followed byDisallow: /. That pair blocks the crawler from the whole site. - Look for
User-agent: *followed byDisallow: /. The star means every crawler, so this blocks all of them, Google included.
A file that blocks ChatGPT’s search crawler looks like this:
User-agent: OAI-SearchBot
Disallow: /
If the page says “not found”, there is no file, and crawlers treat the whole site as open. That’s fine.
Step 2: check whether Cloudflare is blocking AI bots
Cloudflare sits in front of a large share of websites to speed them up and keep out attacks. In July 2025 it began blocking AI crawlers by default and asking every new domain whether to allow them (Cloudflare). Its defaults have changed since, so if your website was set up through Cloudflare, or by someone who uses it, check rather than assume.
A Cloudflare block stops a crawler before it reaches your site, so a clean robots.txt doesn’t prove the bots get in. Ask whoever manages your domain whether it runs through Cloudflare. If it does, the setting to check is AI Crawler and Agent access policies, under Security, then Settings, in the Cloudflare dashboard as of November 2026. It sets a separate rule for search, agent and training crawlers; search should be allowed (Cloudflare docs).
Step 3: check your website builder’s settings
Some website builders have their own way to block AI crawlers. Two common ones, as of November 2026:
- Squarespace has a checkbox, Block known artificial intelligence crawlers, under Settings, then Crawlers. It’s off unless someone turned it on. It blocks training crawlers such as
GPTBot,ClaudeBotandGoogle-Extended, and leavesOAI-SearchBot,PerplexityBotandClaude-SearchBotopen, so it doesn’t keep you out of ChatGPT’s answers (Squarespace). - Wix has no switch; blocking is done by editing
robots.txtin its Robots.txt Editor (Wix). Step 1 covers what to look for there.
On any other builder, search its help center for “AI crawlers” and check which bots its setting names.
Step 4: put the important facts in text
A crawler that gets in can only use what it can read. Microsoft’s guidance for appearing in AI answers warns against putting key information only in images, hiding it in tabs or expandable menus, or relying on PDFs for it (Microsoft Advertising). Your services, the towns you cover, your phone number and your license should be written on the page, not only in a photo of your truck or a downloadable flyer.
Should you block the training crawlers?
For most trade businesses, no. Your website is marketing: it exists to be read and repeated, and letting the training crawlers in is how the assistants come to know your name, your trade and your area.
If you’d rather keep your content out of training, block GPTBot and ClaudeBot and leave the search crawlers open; you stay eligible for ChatGPT’s and Claude’s answers. Think twice about Google-Extended: because it also covers grounding in the Gemini app, blocking it can keep your pages out of Gemini’s answers, not just its training.
What to send your web person
Copy this and send it to whoever manages your website:
Please make sure our website lets these crawlers in: Googlebot, Bingbot,
OAI-SearchBot, Claude-SearchBot and PerplexityBot. Check robots.txt,
Cloudflare's AI crawler settings if we use Cloudflare, and any AI crawler
setting in our website builder.
If they want a robots.txt that allows everything, this is the whole file (with your own address in the last line):
User-agent: *
Allow: /
Sitemap: https://yourcompany.com/sitemap.xml
Once the crawlers can get in, see what the assistants say about you with the steps in How to check whether ChatGPT recommends your business.
Frequently asked questions
Does blocking GPTBot keep me out of ChatGPT's answers?
No. GPTBot collects training data. ChatGPT's search answers come from OAI-SearchBot, which you can allow while blocking GPTBot.
How long after I unblock a crawler will it read my site?
OpenAI says changes to robots.txt for OAI-SearchBot take effect in about 24 hours. Being read again doesn't guarantee being named; that depends on what your site and listings say.
Does Google's AI use a separate crawler?
No. Google says a page only has to be indexed in Google Search, which Googlebot crawls, to appear in AI Overviews or AI Mode. Its separate Google-Extended setting covers Gemini's training and grounding, and Google says it doesn't affect Search.
