Content licensing firm TollBit published first half 2026 data showing that roughly 15% of identified AI page fetching agents reached URLs that robots.txt told them not to, across European sites carrying disallow rules. The finding was reported on 15 August 2026, and the headline number is less useful than the split underneath it: some agents landed on the wrong side of the instruction on about half the sites that named them, while others stayed under 15%.
PPC Land covered the TollBit findings on 15 August 2026. The practical lesson for anyone running a site is not that AI companies are lawless. It is that robots.txt was never an enforcement mechanism. It is a published request, honoured by convention, and the convention is now being interpreted differently by different companies in writing.
What TollBit measured in the first half of 2026
TollBit looked at sites that carry robots.txt disallow rules and counted how often identified page fetching agents ended up on a URL those rules excluded. Across European sites, about 15% of the identified fetchers did so. That is a population level figure covering many agents and many sites, which is why the per agent breakdown matters more than the average.
Three agents stand out. ChatGPT-User, Bytespider and Youbot each reached disallowed pages on roughly 50% of the European sites that explicitly named them in robots.txt. Naming an agent in robots.txt is the strongest signal a publisher can send short of blocking at the server, and on half the sites that sent it, these three arrived anyway. TollBit found ChatGPT-User affected the most sites overall, which follows from how many people use ChatGPT rather than from anything unique about the crawler.
Two agents behaved noticeably better, and behaved differently by region. Claude-User reached disallowed URLs on 9% of European sites against 26% in North America. Perplexity-User was at 13% in Europe against 26% in North America. The same agent, the same instruction, materially different outcomes on either side of the Atlantic. TollBit's data does not explain why the gap exists, and neither will anyone honestly reading it. Regulatory pressure in Europe is one plausible reading. Different site configurations are another. The measurement records the difference without accounting for it.
The table below sets out the per agent figures TollBit reported. Where a regional figure was not given for an agent, it is marked as not stated rather than filled in.
| Agent | Europe | North America |
|---|---|---|
| ChatGPT-User | About 50% of sites that named it, and the most sites affected overall | Not stated |
| Bytespider | About 50% of sites that named it | Not stated |
| Youbot | About 50% of sites that named it | Not stated |
| Claude-User | 9% of sites | 26% of sites |
| Perplexity-User | 13% of sites | 26% of sites |
Robots.txt is a request, not a fence
The file at the root of your domain does not block anything. It publishes a preference and relies on the requesting party to read it and comply. There is no authentication, no rejection at the network layer, and no error returned to a client that ignores it. A crawler that fetches a disallowed URL gets exactly the same 200 response as a browser would.
This has always been true and it has mostly not mattered, because the crawlers that drove meaningful traffic had a commercial reason to behave. What changed is that a new class of fetcher appeared with a different relationship to your site. It is not building an index that sends you visitors. It is retrieving your page to answer a question inside somebody else's product, and in at least one documented case, the company operating it has written down that the robots file may not govern that retrieval.
OpenAI's documented exception, and Anthropic's documented compliance
OpenAI's own crawler documentation states that ChatGPT-User visits a page when a ChatGPT user asks a question, and that because a person initiated the request, "robots.txt rules may not apply". That is not a bug report or an inference from log data. It is the operator describing its own behaviour in its published documentation, and it explains most of the ChatGPT-User number in the TollBit data.
The reasoning behind it is that a user driven fetch resembles a person clicking a link more than it resembles a crawler sweeping a site. Whether you accept that framing is a separate question from whether you plan around it. It is written down, so plan around it.
Anthropic takes the opposite documented position. Its documentation states that all three of its bots respect robots.txt, with no user initiated exception carved out. That divergence in writing is the reason two companies producing similar products show different numbers in the same dataset, and it is the single most useful thing in the TollBit report for a site owner deciding what to do next.
The distinction that decides whether blocking costs you traffic
Here is the part that gets expensive when people get it wrong. ChatGPT-User and OAI-SearchBot are not the same thing and do not do the same job. ChatGPT-User fetches a page in response to a live question from a person using ChatGPT. OAI-SearchBot is the crawler that governs whether your pages can appear in ChatGPT search results at all.
Block both and you lose search visibility inside ChatGPT, while the fetching restriction you were actually trying to enforce carries a documented exception that may let the fetch through anyway. You give up the thing that works and fail to stop the thing that does not. That is the worst available outcome, and it is the default result of a broad block written by pattern matching on the word ChatGPT.
The correct starting point is to decide what you are actually trying to prevent. If the concern is your content being used to answer questions without a visit, that is the fetcher. If the concern is training, that is a different set of agents again. If the concern is control over whether you appear in AI search results, blocking the search crawler is the one lever that reliably works, and it works by removing you.
For most commercial sites, appearing in AI answers is closer to an opportunity than a threat, and the work is to be citable rather than to be invisible. That is the whole premise behind generative engine optimisation: if answers are being assembled from pages, you want yours to be the page. Blocking the search crawler is a decision to leave that surface entirely, and it should be made deliberately, not as a side effect of a robots.txt line written in a hurry.
Cloudflare's default change on 15 September 2026
The other date in this story sits at the infrastructure layer. Cloudflare's 15 September 2026 default change will block Training and Agent crawlers on ad carrying pages for new domains, at the network layer. That is a different mechanism from robots.txt in the way that matters most: a network layer block does not ask, it refuses.
Two limits are worth reading carefully. It applies to new domains, so an existing site does not wake up with different behaviour on 15 September. And it applies to ad carrying pages, so it is scoped to a specific publishing model rather than to the whole web. Anyone launching a new domain after that date should know what the default is before they wonder where their AI referrals went.
What to actually check on your own site
Stop treating the robots file as evidence of anything. It records what you asked for. Your server logs record what happened. Those are different documents and only one of them is a measurement.
Pull your access logs and group requests by user agent. Look for the named agents in the TollBit data and for anything else identifying itself as an AI fetcher. Then cross reference the URLs those agents hit against your disallow rules. If a disallowed path shows up, you now know your robots file is not doing what you assumed, and you can decide whether to escalate to a real block. This is the same log level discipline that a proper technical SEO audit runs on any crawl problem, applied to a newer class of client.
Check your robots file for the specific mistake described above. If OAI-SearchBot is disallowed, confirm that somebody meant to do that. A surprising number of these lines get added defensively and never reviewed.
Decide your position per category rather than per company. Training, agentic fetching for live answers, and search indexing are three different uses of your content with three different consequences for your traffic. A policy that treats them as one thing will be wrong in at least two directions. If you want to be found and cited in AI answers, the constructive half of the work is structural and on page, which is what AI SEO covers.
Finally, do not read a robots directive as a legal instrument. TollBit's numbers show what compliance looks like in practice, and in practice it varies by operator, by region and by agent. If a page genuinely must not be retrieved, the control has to sit at the server or the network, not in a text file that politely asks.
What this means for Thai marketers
The TollBit data covers Europe and North America. It says nothing about Thailand or any other APAC market, and no comparable regional figure for Thai sites exists in this report. Do not assume the European numbers describe what is happening on your logs in Bangkok, in either direction.
What does transfer is the method and the documented behaviour. The OpenAI and Anthropic documentation is global, not regional, so the ChatGPT-User exception applies to a Thai site the same way it applies to a French one. The regional variation TollBit found sits in outcomes, not in policy, so the safe assumption for a Thai site is that you do not know your own numbers until you look at your own logs.
The second thing that transfers is the OAI-SearchBot lesson. Thai sites running bilingual content have more to lose from an accidental block than most, because a Thai language page that gets cited in an AI answer is competing in a much thinner pool than an English one. Removing yourself from that pool by mistake is an expensive way to feel protected.
Frequently asked questions (FAQ)
Does blocking ChatGPT-User in robots.txt actually stop it?
Not reliably, based on both the data and OpenAI's own documentation. TollBit found ChatGPT-User reached disallowed pages on about 50% of the European sites that named it, and OpenAI documents that because a person initiated the request, robots.txt rules may not apply. A server or network level block is the only way to be sure.
Should I block OAI-SearchBot too?
Only if you have decided you do not want to appear in ChatGPT search results, because that is what it controls. OAI-SearchBot is the crawler behind ChatGPT search visibility, not the one that fetches pages for live answers, so blocking it costs you presence without solving the fetching concern.
Does this report say anything about Thailand?
No. The TollBit figures cover European and North American sites only, and nothing in the report addresses Thailand or the wider APAC region. The documented crawler policies from OpenAI and Anthropic are not regional, but the compliance percentages are, so a Thai site owner has no published benchmark to compare against.
How do I find out which AI agents are hitting my site?
Read your server access logs and group the requests by user agent string. That gives you the real record of what was fetched, which robots.txt cannot give you because it only records what you asked for. Cross reference the URLs against your disallow rules to see whether the instruction is being honoured.
Does the Cloudflare change on 15 September 2026 affect my existing site?
No, as described it applies to new domains, and only to ad carrying pages. The change makes blocking Training and Agent crawlers the default at the network layer for those new domains, which is a stronger control than robots.txt because it refuses the request rather than asking the client to withdraw it.
Where to start
Open your logs before you open your robots file. The TollBit numbers are useful as a warning that the instruction and the outcome are two different things, but the only figures that describe your site are the ones in your own access records. Once you know which agents are arriving and what they are taking, the policy question becomes answerable instead of theoretical. If you want help reading those logs and deciding where your content should and should not appear in AI answers, we are glad to work through it with you.
