How many UK websites block ChatGPT search? We checked 5,000

We read the robots.txt files of the 5,000 most-visited .uk websites. One in 14 shuts out ChatGPT's search crawler. Most are newspapers that mean to, and a few dozen look like mistakes.

, 7 min read

The survey in one picture

Skip to the article

  1. 5,000 of the most-visited websites on .uk addresses. Each dot is one of them.

  2. 3,375 had a robots.txt we could read: the file that tells bots which pages they may visit. The other 1,625 had none we could use.

  3. 237 block ChatGPT's search bot, so they can't appear in ChatGPT search answers. That's 1 in 14.

  4. 281 more block OpenAI's training bot but let its search bot in. They opt out of training and stay in ChatGPT search.

  5. Nearly 1 in 5 of the top 100 sites block ChatGPT search: 12 of the 64 with a robots.txt. Further down the list it's 158 of 2,720, about 1 in 17.

  6. 141 of the 286 sites that block an AI search bot belong to five publishing groups, which use one robots.txt for every title.

  7. Some look like mistakes. 18 sites shut AI search out with allow-lists written before it existed. 7 say search=yes, then block it. 4 explain they're blocking training and block search too.

ChatGPT, Claude and Perplexity each send their own crawler to read a page before they can cite it in an answer. If a site's robots.txt tells that crawler to stay away, the site drops out of that assistant's answers. OpenAI says so in its crawler documentation: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." [1]

We wanted to know how many UK websites do this, and whether they mean to.

What we checked

  • The 5,000 most-visited websites on .uk domains, taken in rank order from the Tranco list of 24 October 2026 (list GQ32K). [2]
  • We fetched each site's robots.txt on 25 October. 3,375 returned a usable file. 496 had none, 487 refused our request and 640 didn't answer.
  • For every file we asked one question per crawler: may this bot fetch the homepage? We matched crawler names the way Google does and checked 17 of them, including Googlebot and Bingbot as controls.

Every percentage below is a share of the 3,375 sites with a robots.txt.

The numbers

7.0% of UK sites block OAI-SearchBot, the crawler behind ChatGPT search. That's 237 sites, and 202 of them still let Googlebot in: they stay in Google while keeping out of ChatGPT search.

Training crawlers are blocked about twice as often. 15.3% block GPTBot, the crawler OpenAI uses to gather training data, and 281 sites block GPTBot while leaving OAI-SearchBot open. Those sites have opted out of training and kept their place in ChatGPT search, which is the split OpenAI's crawler names are designed to allow.

Share of 3,375 UK sites whose robots.txt blocks each crawler from the homepage
  1. GPTBot 15.3%, AI training
  2. CCBot 15.3%, AI training
  3. ClaudeBot 14.6%, AI training
  4. Google-Extended 9.9%, AI training
  5. PerplexityBot 7.6%, AI search and user requests
  6. ChatGPT-User 7.1%, AI search and user requests
  7. OAI-SearchBot 7.0%, AI search and user requests
  8. Claude-SearchBot 5.8%, AI search and user requests
  9. Perplexity-User 3.4%, AI search and user requests
  10. Googlebot 1.0%, Google search (control)
Show the numbers
Crawler or groupValueType
GPTBot 15.3% AI training
CCBot 15.3% AI training
ClaudeBot 14.6% AI training
Google-Extended 9.9% AI training
PerplexityBot 7.6% AI search and user requests
ChatGPT-User 7.1% AI search and user requests
OAI-SearchBot 7.0% AI search and user requests
Claude-SearchBot 5.8% AI search and user requests
Perplexity-User 3.4% AI search and user requests
Googlebot 1.0% Google search (control)

Bigger sites block more. 12 of the 64 sites in the UK top 100 that have a robots.txt block OAI-SearchBot. Further down the list the share falls quickly.

Share of UK sites blocking OAI-SearchBot, by traffic rank among .uk sites
  1. Top 100 18.8%
  2. 101 to 1,000 11.3%
  3. 1,001 to 5,000 5.8%
Show the numbers
Crawler or groupValue
Top 100 18.8%
101 to 1,000 11.3%
1,001 to 5,000 5.8%

The type of domain matters too. 8.8% of .co.uk sites block OAI-SearchBot, against 2.5% of universities on .ac.uk and 1.7% of government sites on .gov.uk.

Most of it is newspapers, on purpose

Of the 286 sites that block at least one AI search crawler, 141 belong to five publishing groups that use one robots.txt across every title: 84 Newsquest local papers, 38 Reach titles including the Mirror and the Manchester Evening News, eight titles from one regional group, six DMG Media sites including the Daily Mail and Metro, and five Condé Nast magazines.

Look closer and the files read like business decisions about which AI companies get in.

  • The BBC blocks OAI-SearchBot, ChatGPT-User and PerplexityBot by name, alongside the training crawlers.
  • The Sun starts with User-agent: * and Disallow: /, then lets back in Googlebot and OpenAI's three crawlers. Claude and Perplexity stay blocked because they aren't on the list.
  • The Guardian does the opposite: it blocks Claude-SearchBot and PerplexityBot by name and doesn't mention OpenAI's crawlers at all.

Where it looks like a mistake

A few dozen files block AI search in a way that looks unintended or contradicts itself.

Allow-lists written before AI search existed

18 sites block every crawler with User-agent: * and then let named search engines back in. AI search crawlers aren't on the list, so they're shut out. Pinterest UK's file does this: Googlebot gets its own group of rules, and line 999 of the 1,035-line file reads User-agent: * followed by Disallow: /.

Files that say yes, then no

Seven sites publish Cloudflare's content signal search=yes and further down the same file disallow AI search crawlers. One UK software supplier's file declares Content-Signal: search=yes,ai-train=no and later has a section headed AI Search crawlers that blocks OAI-SearchBot.

Comments that only mention training

Four files explain in a comment that they're blocking AI training, while the rules underneath also block search crawlers.

Crawler names that are no longer used

415 files name anthropic-ai and 374 name Claude-Web. Neither appears in Anthropic's current list of crawlers, which names ClaudeBot, Claude-SearchBot and Claude-User. [3] 20 sites block anthropic-ai and leave ClaudeBot allowed, so they haven't blocked the crawler they were probably aiming at.

Rules that block everything

35 files block Googlebot as well. DPD UK's file lists dozens of folders under User-agent: * and ends with Disallow: /**. Under Google's matching rules that last line covers every URL on the site, for every crawler.

Blocking training doesn't have to mean leaving AI search

Each AI company now uses separate crawler names for training, for search and for fetching a page when a user asks, so you can block one without the others.

OpenAI uses GPTBot to gather training data, while OAI-SearchBot decides whether you can appear in ChatGPT search. ChatGPT-User fetches pages a user asks for, and OpenAI says "robots.txt rules may not apply" to it because a person started the request. Changes take about 24 hours to reach ChatGPT search. [1]

Anthropic's ClaudeBot collects training data, Claude-SearchBot indexes pages for Claude's search results and Claude-User fetches pages for people using Claude. [3]

Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models". Perplexity-User "generally ignores robots.txt rules" because a user requested the fetch. [4]

Google-Extended controls whether your content trains Gemini and grounds some of Google's other AI products, and Google says it "does not impact a site's inclusion in Google Search". [5] AI Overviews and AI Mode follow your Googlebot rules. [6] 300 of the sites we checked block Google-Extended and leave Googlebot open, which keeps them in Google's AI answers.

A robots.txt that opts out of training and stays in AI search names the training crawlers and leaves the rest to the default:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

OAI-SearchBot, Claude-SearchBot and PerplexityBot match the User-agent: * group, so they're allowed.

robots.txt isn't the only gate

46% of the responses we received came through Cloudflare, and Cloudflare can block crawlers before they ever read robots.txt. On 1 July 2026 it split its AI crawler controls into "three major use cases: Search, Agent, and Training crawlers". [7] On 15 September it changed them again and said "Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot". The same post says "less than 1% of Cloudflare sites choose to block Search bots", while 17% block training. [8]

We can't see those settings from outside, because they act on the crawler's verified identity. Google's advice covers both layers: make sure crawling "is allowed in robots.txt, and by any CDN or hosting infrastructure". [6]

How to check your own site

  1. Open /robots.txt on your domain. Search it for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot, and for a User-agent: * group containing Disallow: /.
  2. If a rule mentions a crawler name you don't recognise, check it against the company's current documentation. Lists copied from old blog posts carry retired names.
  3. If your site runs through Cloudflare or another CDN, check its bot settings for the search category as well as robots.txt.
  4. Allow a day for changes to reach ChatGPT search.

Method and limits

  • We read robots.txt only and judged each crawler against the homepage path. A site can allow the homepage and block deeper sections.
  • Blocks set in a CDN or firewall don't show in robots.txt, and we didn't pretend to be AI crawlers to test them.
  • Our crawler identified itself as an aeo.blog survey. The 487 sites that refused it may treat real search crawlers differently.
  • We fetched the bare domain, falling back to www. 177 domains redirected to a different site, and we read the file we ended up on.
  • Crawler names were matched exactly, as Google does. A looser parser disagreed on 26 sites.

Sources

  1. OpenAI, Overview of OpenAI crawlers
  2. Tranco list GQ32K, created 24 October 2026
  3. Anthropic, Does Anthropic crawl data from the web, updated 7 April 2026
  4. Perplexity, Perplexity crawlers
  5. Google, Google's common crawlers, updated 15 October 2026
  6. Google, AI features and your website, updated 10 December 2025
  7. Cloudflare, 1 July 2026
  8. Cloudflare, 15 September 2026

Is your website in the answers?

The SEO & AEO Checker on Raq.com answers three questions about your own site.

Check your site