TL;DR
- Cloudflare said on 28 August 2026 that BotBase now gives bot operators submission tracking, editing and an automated review pipeline, built after annual new bot submissions rose roughly seven times since 2023.
- Submissions now carry one of three statuses: Waiting for review, Accepted, or Rejected with a stated reason and steps to fix before resubmitting.
- The history record starts at the 28 August launch, and submissions made before that date are not described as being backfilled into the view.
- Operators now declare behaviour, content use in the Content Signals vocabulary, and whether they are direct or intermediary, replacing a single category label.
- Cloudflare restates that the decision on what traffic is admitted rests with each individual site owner, and gave no figures on acceptance or rejection rates.
Cloudflare published a post on 28 August 2026 opening the operator side of BotBase, the directory of known bots and agents it introduced two months earlier, adding submission tracking, editing and an automated review pipeline. According to PPC Land, which reported on the announcement written by Julian Laxman on the Cloudflare Blog, the company built the pipeline after the annual volume of new bot submissions rose roughly seven times since 2023. This reads as an operator release, and it is worth publisher attention for one reason: the pipeline it formalises is what decides which crawlers arrive at a site with a verified identity attached.
What changed on 28 August
Cloudflare said on 28 August 2026 that the bot submission experience now has three parts rather than one form. There is a bots directory for browsing, searching and filtering the catalogue Cloudflare already tracks, which is the same set exposed publicly through Cloudflare Radar. There is a submission form for new entries. And there is a submission history tab that tracks everything an account has sent.
The third element is the substantive one. Until 28 August, a company running a crawler could fill in the form and then wait indefinitely, with no record of what happened next. Support email was the only route to an answer. Cloudflare described hearing consistent feedback from bot operators that submitting a bot felt like a black box: a form is filled in, submit is pressed, and nothing observable follows.
The form also moved. It previously sat under Manage Account and then Configurations, a path that tied a bot to a billing account. It now sits under Protect and Connect, then Application Security, then BotBase, next to the trust and bot tooling operators already use. Access is open to all customers directly from the Cloudflare dashboard.
Why this is publisher news, not just operator news
BotBase itself arrived on 1 July 2026, alongside policy changes in which Cloudflare replaced its binary framing of AI bots with a taxonomy built on behaviour rather than label. That release gave website owners a searchable database of tracked crawlers, and separately a dashboard measuring how often each operator fetched a page against how many readers it returned. What it did not give was any corresponding visibility to the companies on the other end of the connection.
The 28 August release closes that asymmetry, and in doing so it makes the directory more accurate. Cloudflare frames accuracy as functional rather than cosmetic: current details are described as a key component of how a bot earns and keeps Verified status, and the company says that status increasingly determines whether sites across its network can allow a crawler based on its behaviour. If the directory is the input to that decision, then anything that improves the directory changes what reaches a site.
Cloudflare restates the limit on that mechanism, and the limit is important. The decision on what traffic is admitted rests with each individual site owner. Cloudflare classifies; the site owner allows or blocks. That division is the reason a publisher should read this release as information rather than as a policy that has been applied on their behalf.
Three statuses replace an empty queue
What each status means
Submission history now assigns every entry one of three states. Waiting for review means the submission has been received and sits in the queue. Accepted means the bot is tracked in the directory. Rejected means something in the submission requires change, and Cloudflare says it supplies the reason together with steps that can be acted on before resubmission.
Opening an individual submission exposes its full detail. A rejected entry shows why it failed. An accepted entry that Cloudflare reclassified during review shows what was altered, which matters because classification determines how sites treat the crawler afterwards. An operator who submitted a crawler as one thing and had it recorded as another can now see that happened, rather than discovering it through the traffic outcome.
The boundary on historical submissions
One limit is stated plainly and deserves attention from anyone auditing an existing entry. The record covers every bot submitted from the account starting from the launch itself. Submissions made before 28 August 2026 are not described as being backfilled into the history view. An operator who submitted a crawler in July and is waiting on it should not read an empty history as evidence that the submission was lost, and should not read it as evidence that it is progressing either. A separate filter labelled My bots, available from the bots directory screen, surfaces all bots submitted under the account currently logged in.
Editing, and why a directory needs it
Identification details drift. A crawler's operator might rehost an IP list at a new endpoint after a site redesign, or migrate from an IP allowlist to signing traffic with Web Bot Auth. Before this release, reflecting either change meant completing the entire form again and creating a brand-new entry, with the duplication that implies for a directory meant to be authoritative. Editing an existing submission is now possible, and a submission still waiting for review can be cancelled.
Web Bot Auth is the cryptographic scheme Cloudflare has been building since May 2025, formalised through a registry format for bot and agent authentication published on 30 October 2025 in partnership with Amazon Bedrock AgentCore. For a site owner the practical relevance is that identity claims backed by a signature are harder to spoof than identity claims backed by a user agent string, which is the failure mode most publisher bot policies were built around.
Three declarations replace a single label
The intake form was rebuilt on the behaviour and content use model Cloudflare introduced on 1 July 2026. Instead of compressing a crawler into one category, an operator now declares three separate things.
Behaviour
A bot might index pages for search, act as an agent on a person's behalf, collect data, train models, or support SEO tooling. Multiple behaviours can be selected rather than only the closest approximation, which addresses the mixed-purpose crawler problem directly. That problem is not hypothetical. PPC Land reported on 21 August 2026 that Cloudflare had set four disclosure requirements determining whether a mixed-purpose crawler avoids being blocked on sites that disallow training.
Content use
The second declaration covers what the crawler retains and reshares after access. Cloudflare distinguishes a bot skimming a page for a search snippet from one storing the same page to train a model, and asks operators to declare a level using the same Content Signals vocabulary website owners already apply to their own files. The post gives a worked example of what a site might publish in robots.txt: Content-Signal: search=yes, ai-train=no, use=reference, permitting indexing and a stored reference while refusing model training. An operator's declaration is checked against exactly that kind of preference.
Who runs the bot
An operator crawling the web from its own infrastructure to build its own index is classified as direct. A platform carrying traffic that other companies decided to send is an intermediary. The post illustrates the distinction with a general-purpose AI assistant fetching a page because a person typed a question into a different company's application built on that assistant's API. The assistant operator supplies the infrastructure, but the request originated in someone else's product.
The three declarations side by side
The table below sets out what an operator declares under each of the three fields Cloudflare described, and what a site owner can do with that information.
| Declaration | What the operator states | What it tells a site owner |
|---|---|---|
| Behaviour | Indexing for search, acting as an agent for a person, collecting data, training models, or supporting SEO tooling; multiple selections allowed | Whether one crawler serves more than one purpose, so a policy can be written per purpose rather than per company name |
| Content use | What is retained and reshared after access, declared in the Content Signals vocabulary such as search=yes, ai-train=no, use=reference | Whether the crawler's stated use matches the preference the site already publishes in its own file |
| Who runs the bot | Direct, meaning crawling from its own infrastructure for its own index; or intermediary, meaning carrying traffic other companies decided to send | Whether a fetch represents an index being built or a request that originated in an end user's question inside another product |
Why behaviour plus content use beats a single label
A single label forces a decision no publisher actually wants to make. Blocking a company blocks every purpose that company's crawler serves, including the ones the publisher wanted. Allowing it allows every purpose too. A crawler that indexes for search and also collects training data has, under a single-label scheme, to be recorded as whichever purpose the operator judged closest, and the site owner then writes policy against a description that was already lossy before it reached them.
Separating behaviour from content use makes the two decisions independent. Indexing and reference use are what get a page cited in AI answers. Model training is a separate question, and refusing it does not remove the page from the systems that cite it. Treated as one on/off switch, those collapse into a choice between visibility and control. Treated as two declarations, a publisher can permit the first and refuse the second, which is what the worked example in the post expresses: search=yes and use=reference alongside ai-train=no.
That is the practical reading for anyone whose traffic strategy now includes being surfaced inside generated answers rather than only in a list of blue links. The mechanics of that are covered in more depth in the material on generative engine optimisation and on AI-era SEO, but the policy decision itself is small and specific: decide the retention question and the citation question separately, then write the file accordingly.
Direct and intermediary are different problems
The operator type declaration matters more than it first looks. A direct crawler fetching pages to build an index behaves like the crawlers publisher policies were written for. It arrives on a schedule, it fetches broadly, and its value exchange is legible: it takes pages and it may send readers.
An intermediary fetch is different in kind. It happens because a person asked a question somewhere else, in an application built on another company's API, and the fetch is a step in answering that person. It is not building an index and it may never return. Blocking it does not protect content from an index, it removes the page from an answer a real person is waiting for. Allowing it does not guarantee a referral either. Neither outcome is obviously right, and the point is that a publisher cannot even weigh them until the fetch declares which kind it is. Before this taxonomy, both arrived as an IP address and a user agent string.
There is also an attribution consequence. If intermediary traffic answers a question without a click, then the count of sessions in analytics understates the reach of the page, and that gap widens as more questions are answered inside assistants. That is a measurement problem rather than a blocking problem, and it argues for tracking brand mentions and citations alongside sessions, which is the same argument running through work on visibility inside AI assistants.
What this means for Thai marketers
For a Thai publisher or brand site, the first action is small and free: read what the site currently publishes in robots.txt, and check whether it expresses a content use preference at all or only an allow and disallow list. Most Thai sites are still on the second, which was the only vocabulary available until recently. Adding a Content Signals line does not block anything by itself, and it does state a preference that an operator declaration can be checked against.
The second action is deciding the two questions separately rather than together. For most Thai commercial sites, indexing and reference use are worth keeping, because that is what puts a page inside an answer when someone asks an assistant which supplier or clinic or agency to use in Bangkok. Model training is the question a site can answer with no if it wants to, without giving up the first. Sites that block everything to be safe are frequently blocking the mechanism that would have cited them, which is a more expensive mistake than the one they were avoiding, and it usually shows up months later as an absence rather than as a drop.
Third, note the limits honestly. Cloudflare's classification governs traffic across Cloudflare's network, and the final allow decision still sits with the site owner, so a Thai site not on Cloudflare gains nothing automatically from any of this. Nothing in the announcement is specific to Thailand or to Asia, and no figures were given for how many submissions are accepted or rejected. What the release changes is the quality of the identity information available when a decision is made, which is worth folding into the same review cycle as the rest of a site's organic search work rather than treated as a separate security project.
What the source does not say
Several things are absent from the announcement, and they bound how far any of this can be read. Cloudflare gave no figures for how many submissions are accepted and how many are rejected, so the acceptance rate is unknown. It gave no absolute submission counts either. The only quantity stated is the growth: annual volume of new bot submissions rose roughly seven times since 2023, which describes a trend without establishing a base.
The post also does not describe how the automated review reaches a decision. Whether it applies a fixed rule set, checks declarations against observed behaviour, or escalates ambiguous cases to a person is not stated, and that matters because a rejection reason is only as useful as the process that produced it. Nothing in the announcement is specific to Thailand or Asia, and nothing addresses how existing entries submitted before 28 August 2026 will be reviewed against the newer three-declaration model.
Frequently asked questions
Does a submission made before 28 August 2026 appear in the new history tab?
No, based on what Cloudflare published: the record covers every bot submitted from the account starting from the launch itself, and submissions made before 28 August 2026 are not described as being backfilled into the history view. An operator waiting on an older submission should treat an empty history as an absence of information rather than as a status.
What do the three submission statuses mean?
Waiting for review means the submission has been received and sits in the queue, Accepted means the bot is tracked in the directory, and Rejected means something requires change before resubmission. Cloudflare says a rejection comes with the reason plus steps that can be acted on, and an accepted entry that was reclassified during review shows what was altered.
Does refusing AI training remove a page from AI answers?
Not according to how the Content Signals vocabulary is structured, because retention for training and use as a cited reference are separate declarations. The worked example in Cloudflare's post is Content-Signal: search=yes, ai-train=no, use=reference, which permits indexing and a stored reference while refusing model training.
Does Cloudflare decide which bots reach a site?
No. Cloudflare classifies crawlers and maintains Verified status, and the company restates that the decision on what traffic is admitted rests with each individual site owner. The classification is an input to that decision across Cloudflare's network, and a site not behind Cloudflare is not covered by it at all.
If the site's robots.txt still says nothing about content use, that is the one file worth opening this week. Decide the citation question and the training question separately, write both down, and the operator declarations Cloudflare is now collecting have something concrete to be checked against.
