IAB Australia published a bots and crawler decision framework on 28 July 2026 that treats crawler policy as five separate decisions, not one. Written by Jonas Jaanimagi, the industry body's technology lead, it sorts every crawler into one of five categories and asks the site owner to give each category one of four verdicts: allow, allow with conditions, require licensing, or block. The framework is advisory, so it obliges nobody.
Blocking AI bots is not one decision
The idea worth carrying away from the 28 July 2026 guidance is that "block AI bots" covers several different actions a site owner rarely wants to take together. Googlebot indexing a page for search, GPTBot collecting text for model training and AdsBot-Google checking a landing page for ad quality are separate crawlers doing separate jobs, and IAB Australia puts them in separate categories. A blunt rule aimed at one can stop the other two. A site can keep indexing and ad serving open while licensing or refusing training access, but only if the rule names user agents.
The five crawler categories in the IAB Australia matrix
The framework published on 28 July 2026 sorts every crawler into one of these five buckets before any allow or block decision.
| Category | Example user agents | What it does |
|---|---|---|
| A. Discovery and search indexers | Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, Applebot | Indexes pages so they can surface in search results |
| B. AI training crawlers | GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider | Collects content used to train models |
| C. Live AI agents and user fetchers | ChatGPT-User, Claude-User, Google-Agent, Perplexity-User | Fetches a page in real time when a person asks something |
| D. Operational and advertising infrastructure | AdsBot-Google, Mediapartners-Google, verification fetchers, platform link-preview bots | Runs ad serving, verification and link previews |
| E. Impostors and unknowns | Spoofed user agents, python-requests, residential proxies, undocumented crawlers | Unverifiable identity, no stated purpose |
Four verdicts, assigned per category
IAB Australia's 28 July 2026 guidance then asks the site owner to give each category a verdict.
- Allow, where the value justifies leaving access unrestricted.
- Allow with conditions: rate limits, path scope or attribution terms.
- Require licensing: access under a commercial agreement only.
- Block, where there is no value, a real cost, or an identity that cannot be verified.
Category E usually ends at block, because an unverifiable identity gives you nothing to weigh. Categories A and D are where a careless block does the most damage, since those crawlers keep a page indexed and its ads serving.
Where the statistics in the guidance come from
IAB Australia aggregates other people's measurement; it does not run its own. It names its sources as DataDome's AI Traffic Report for Q2 2026, Cloudflare Radar readings from mid-2026, HUMAN Security's 2026 State of AI Traffic, Gartner and Adobe Digital Insights.
Cloudflare's mid-2026 data, as quoted in the guidance, puts 57.5% of web page requests as automated, the first recorded crossover past human traffic. Cloudflare also puts roughly 52% of AI crawler requests as targeting model training and about 2.6% as real-time, human-triggered fetches. DataDome's numbers, again as quoted by IAB Australia, put 80% to 88% of AI referral traffic as originating from ChatGPT despite declining crawl volume, and AI agent requests growing 45% quarter on quarter to 17.7 billion in Q2 2026. Both companies sell bot management and edge services and measure traffic crossing their own networks, worth stating whenever the percentages are repeated.
The 15 September 2026 date attached to Cloudflare defaults
The guidance carries one dated instruction: before 15 September 2026, review your AI crawler defaults if the site runs on Cloudflare or managed edge or WAF infrastructure, because new defaults will restrict training and agent bots on pages that display ads. That is a change to configuration, not content, and it can land without anyone on the marketing side touching a setting.
What this means for Thai marketers
The guidance is written for Australian publishers and does not mention Thailand. The user agents and the Cloudflare policy are global, though, so a Thai site behind Cloudflare falls under the same default change on 15 September 2026. If your site displays ads and sits behind Cloudflare, that review belongs on a calendar before the date.
The rest is a decision to make on purpose rather than inherit from a default. If visibility inside AI answer engines matters commercially, blocking category B training crawlers carries a cost, and refusing category C live fetchers carries a more immediate one, because those requests come from a person asking something right then. If the traffic model depends on search, categories A and D stay open whatever you decide about the rest. Auditing which bots reach your pages belongs with the rest of a technical SEO programme, and the trade-off between training access and citation visibility is the central question in AI search optimisation.
Frequently asked questions
Does blocking GPTBot also block Google?
No. GPTBot and Googlebot are separate user agents in separate categories of IAB Australia's 28 July 2026 matrix, so a rule naming GPTBot leaves Googlebot untouched. A blanket rule against all crawlers is what puts search indexing at risk.
Will Google penalise a site that blocks AI training crawlers?
The guidance does not say. It sets out categories and verdicts and makes no claim about how Google treats a site that blocks category B crawlers, so anyone citing it for that is going beyond the source.
Does the 15 September 2026 date apply to Thai websites?
Yes, if the site runs on Cloudflare or managed edge or WAF infrastructure, because that policy is global, not Australian. The guidance is written for Australian publishers and does not mention Thailand, so treat the date as a configuration deadline, not local regulation.
Who measured the traffic figures the guidance quotes?
Cloudflare and DataDome measured them, not IAB Australia. The 57.5% automated-traffic figure and the 52% training versus 2.6% real-time split come from Cloudflare Radar in mid-2026; the 80% to 88% ChatGPT referral share and the 17.7 billion agent requests in Q2 2026 come from DataDome.
Is the IAB Australia framework mandatory?
No. It is advisory guidance from an industry body and carries no obligation, so its value is in the structure it gives the decision.
The practical work after 28 July 2026 is smaller than the topic sounds: list the crawlers that reach your pages, put each in a category, give the category a verdict, and write the rule by user agent. If you want that mapped against your own traffic before 15 September, a GEO and AI visibility review with the Relevant Audience team is where it starts.
