Cloudflare now writes your robots.txt from your AI crawler policy

Cloudflare published a feature on 21 August 2026 called Bot Preference Sync, which generates or updates a site's robots.txt file directly from the AI crawler policy already configured in the Cloudflare dashboard. It is stated as available on every plan from free to Enterprise, can be switched on or off at any time, and its availability was given as the coming week, with the company's changelog named as the place where the rollout will be confirmed. Alongside it, the Disallow setting for the Training category now writes a no-training preference in a form that lets cooperating mixed-purpose crawlers keep reaching content for search indexing, on the condition that they let site owners verify how the data is used.

The post was written by Jin-Hee Lee and tagged under AI Bots, Bot Management and Product News. PPC Land reported it on 23 August 2026.

The problem it is built to solve: a stated preference and an enforced rule that disagree

Two files have governed how a website states its position on automated traffic, and for years they have been allowed to contradict each other. One is robots.txt, the plain-text declaration a crawler reads before fetching a page. The other is the enforcement layer, meaning firewall rules, bot management settings and edge blocks that act regardless of what any crawler claims to honour. Cloudflare's argument is that the gap between the two has become a liability for the site owner rather than for the crawler.

The failure mode the post describes is specific. A robots.txt file can state that a crawler is disallowed while the enforcement rules quietly let that same crawler through. When the stated preference and the enforced rule disagree, according to Cloudflare, some crawlers treat the inconsistency as a basis to disregard the preference, or to attempt a bypass of the enforced rules. Synchronising the two removes that argument. That is the whole design intent, and it is worth separating from the question of whether any given crawler obeys robots.txt at all, which this feature does not change.

How the file actually gets written

The mechanism is additive rather than destructive, which matters if you already maintain a hand-written file. Where a site already has a robots.txt file, the contents generated by Bot Preference Sync are prepended to the existing material, so any Disallow directives already present survive the operation. The generated section is fenced by comment markers: the example published in the post, shortened and anonymised by Cloudflare, opens with a line reading BEGIN Cloudflare Bot Preference Sync, lists four user agents named TrainingBot1, TrainingBot2, TrainingBot3 and MixedUseBot-Extended, applies a site-wide Disallow, and closes with a matching END marker.

The membership of that user-agent list is not frozen at the moment of configuration. Cloudflare states that it will draw on the bots it tracks in BotBase to periodically refresh the set of user agents written into robots.txt whenever a customer chooses to block or disallow a given category. Verified bots classified as Search, Agent and Training can be inspected at any point through the company's public bots directory. The practical effect is that a preference expressed once against a category keeps covering new crawlers entering that category, without the site owner editing anything.

What it deliberately does not handle

The stated limit is that Bot Preference Sync addresses policy decisions taken across a whole category rather than negotiated crawler by crawler, and that it will not read from individual custom rules containing more complex logic. A customer that has struck a particular arrangement with a named company, and wants an exception carved for that company alone, retains the option of switching the sync off and hand-tailoring the file to match the custom policy.

That is the single most important sentence in the announcement for anyone with a licensing deal or a bespoke allowlist. The sync writes from category-level policy. If your real policy lives in a custom rule with conditions attached, the generated file will not reflect it, and turning the sync on would publish a statement that is simpler than your actual arrangement.

Three categories, and a narrower meaning for Disallow

The taxonomy underneath the feature is the one Cloudflare introduced on 1 July 2026, when it replaced its earlier binary framing of AI bots with three behavioural categories. The table below sets out what each covers and what the August 2026 post says about the settings attached to it.

Training is where the substantive change sits. The reasoning Cloudflare gives for letting those crawlers through is conditional rather than charitable: they keep access because they permit site owners to verify directly how the data is used. Search visibility for cooperating crawlers, according to Cloudflare, is unaffected. The company's position on the underlying problem has not softened either. Its July post argued that mixed-use crawlers, described as "bots that blend search, agent use, and training behind a single user agent," put site owners at a disadvantage precisely because they make it difficult to separate wanted behaviour from unwanted behaviour, and that argument is restated as still standing.

The four disclosure conditions

For the purposes of bot verification, operators whose bots perform both Search and Training must supply additional information in order to avoid being blocked on sites where Disallow Training has been set. Four requirements are listed:

1. The bot must respect a no-training preference in robots.txt, through any mechanism.
2. The operator must give site owners a route to opt out of AI summaries.
3. The operator must provide URL-level visibility into which pages were made available for training, together with metrics on search results, so a site owner can see how content was used on each side.
4. The operator must be able to demonstrate publicly that disallowing training does not damage traditional search results.

Bots from leading AI model developers and service providers that satisfy those criteria are to be tracked publicly in the AI bot transparency section of Cloudflare Radar, which the company says will include cases where best practices are honoured alongside cases where they are not. Crawlers that decline to supply the disclosure receive no benefit of the doubt and remain blocked wherever training is disallowed. Cloudflare frames the arrangement as "making Transparency the price of admission."

The third condition is the one with no equivalent at any major operator today. URL-level reporting on what was taken for training, paired with metrics on search results, would give a publisher a per-page account of what was collected and what came back. Whether any large operator supplies it is an open question, and Cloudflare Radar's transparency section is named as where the answer becomes public.

A different starting position for ad-funded domains

The post also changes what a new domain inherits on day one, and it splits the default in two. At onboarding, a customer can select an option stating that the domain monetises from pages carrying advertising. Selecting it sets Training to Disallow as the default, on the stated expectation that sites depending on advertising revenue reserve those pages for human visitors. The setting is changeable at any later point, and Cloudflare summarises the intended outcome as remaining in search while keeping content out of model training.

For every other new customer, nothing is applied at onboarding. No blocks and no disallows are added, and Search, Agent and Training can each be blocked afterwards at the customer's choosing. For all new customers, Bot Preference Sync itself is switched on by default. Existing customers running the legacy managed robots.txt feature will be prompted to review and confirm their preferences in order to transition to the new system when it launches.

The honest tension for anyone doing GEO

This is the part that does not resolve cleanly, and pretending otherwise would be dishonest. Blocking training crawlers is a defensible commercial decision, and it can also cost AI answer visibility, because a mixed-purpose crawler that does both search and training is exactly where the trade-off bites. That is the situation the Disallow refinement is trying to address: it attempts to make a training refusal separable from a search penalty, but only for crawlers that cooperate and disclose.

The measured cost on the other side of the ledger is on the record. Research from Rutgers Business School and The Wharton School found news publishers that blocked AI crawlers through robots.txt lost roughly 7 percent of weekly website traffic within six weeks, a figure reported elsewhere in the same body of work as 23.1 percent of monthly visits, without proportional protection, because the protocol remains voluntary. Compliance is partial in the other direction too: TollBit measurement reported in August 2026 found 15 percent of AI page fetchers in Europe reaching URLs that had been disallowed, and Microsoft Clarity added robots.txt violation flags inside its Bot Analytics view in June 2026, supplying evidence without enforcement.

Two things follow from that, and neither is a prediction. First, a robots.txt directive is a request, and an edge block is an action, so the two answer different questions and a site owner needs to know which one they are relying on. Second, a training refusal that cooperating crawlers honour without a search penalty only holds for crawlers that cooperate, which is the exact gap the four disclosure conditions are meant to close. Whether operators supply that disclosure is not something the announcement settles, and there is no basis in this post for predicting adoption or for claiming how any AI company will respond.

What a site owner should check

• Whether the dashboard policy and the live robots.txt currently say the same thing, because the feature is built on the assumption that many sites will find they do not.
• Whether the site would be classified as monetising from advertising at onboarding, since that answer now assigns a different training default from every other new domain.
• Whether any negotiated or bespoke crawler arrangement exists, because the sync writes category-level policy and does not read complex custom rules, so turning it on would overwrite the stated position with a simpler one.
• Whether the existing robots.txt contains Disallow directives that must survive, which the prepend behaviour is designed to preserve, and whether the BEGIN and END Cloudflare Bot Preference Sync markers are present after the first sync.
• Whether anyone on the team is still hand-editing robots.txt, because with the sync on, the file becomes an output of a dashboard setting rather than a document a person maintains.

What this means for Thai marketers

What follows is Relevant Audience's reasoning, not a claim from the source, which says nothing about Thailand. Cloudflare sits in front of a large share of the web and this feature ships on the free plan, so the decision reaches small Thai sites that never thought of themselves as having a crawler policy at all. The practical exposure is not the blocking. It is the discovery that a dashboard toggle set months ago is about to be published as a public statement in a file that AI operators, auditors and licensing counterparties can read.

Three groups have different problems here. A Thai publisher running programmatic display inventory should decide deliberately whether the ad-monetisation answer at onboarding matches how they actually want AI training treated, rather than inheriting it. A brand or e-commerce site that wants to appear in AI answers should be careful not to acquire a blanket training refusal by accident while its real goal is retrieval, which is the surface the generative engine optimisation work addresses. And any site that has been relying on a permissive robots.txt with restrictive edge rules, or the reverse, loses that ambiguity unless the sync is turned off deliberately. Crawler access is a technical setting with an editorial consequence, which is why it belongs in the same conversation as AI SEO rather than being left to whoever last touched the firewall.

FAQ

Will Bot Preference Sync delete my existing robots.txt rules?

No, the generated block is prepended to the existing file rather than replacing it, so Disallow directives already present survive the operation. The generated section is fenced between BEGIN and END Cloudflare Bot Preference Sync comment markers so it can be identified in the file.

When exactly does it become available?

Cloudflare gave availability as the coming week rather than a specific date, and named its changelog as the place where the rollout will be confirmed. The post itself was published on 21 August 2026.

Does disallowing training remove a site from AI search results?

According to Cloudflare, search visibility for cooperating crawlers is unaffected, because the Disallow option now writes the no-training preference in a form those crawlers can honour while still indexing for search. That protection applies only to crawlers that meet the four disclosure conditions; crawlers that decline to supply the disclosure remain blocked wherever training is disallowed.

What happens if I have a negotiated deal with a specific AI company?

The feature handles category-level policy and will not read individual custom rules containing more complex logic, so a bespoke arrangement is not reflected in the generated file. Cloudflare's stated answer is that such a customer switches the sync off and hand-tailors robots.txt to match the custom policy.

Does this apply to sites on the free plan?

Yes, the feature is stated as available to all customers from the free tier to Enterprise, and can be turned on or off at any time. For all new customers, Bot Preference Sync is switched on by default, and existing customers on the legacy managed robots.txt feature will be prompted to review and confirm their preferences to transition.