Common Crawl, the nonprofit archive that fetches more than two billion webpages every month and publishes the result for anyone to download, released an 18-page guide in June 2026 called The AI Visibility Audit. It explains how a site owner can check whether their pages are actually present in the dataset that most large language models were trained on. Search Engine Journal covered the guide on 10 August 2026 and made the point that the procedure, as Common Crawl wrote it, is carried out entirely by hand.
The stakes are set out in the same article. Search Engine Journal reported that 64% of large language models released between 2019 and 2023 were trained on Common Crawl data, and that GPT-3 was roughly 60% filtered Common Crawl by training weight. The July 2026 crawl contained 2.14 billion pages. A site missing from that archive was missing from a large share of what those models read during training.
What the Common Crawl audit asks a site owner to do
The guide Common Crawl published in June 2026 walks through a sequence of manual checks. Search Engine Journal summarised the sequence as curling your homepage with CCBot's user agent and comparing the response to a browser's, then using the index API to eyeball your capture counts, then looking up your rank in Common Crawl's web graph, then compiling a scorecard per domain by hand. None of those steps is technically hard. All of them are slow, and the whole routine falls apart the moment you have more than a couple of domains to check.
The recommendations inside the guide are short enough to list in full, and Search Engine Journal reported them in this order. Check CDN settings before touching server files, because that is where most blocking now happens. Allow CCBot in robots.txt, where the absence of a rule already counts as permission rather than a gap. Add sitemaps to robots.txt so the crawler has a discovery path. Render content server-side. Earn links from well-connected sites, because crawl priority follows the link graph.
Why the order of those steps is the useful part
Putting the CDN first is not a stylistic choice. A robots.txt file that says CCBot may crawl is worth nothing if the request never reaches the origin server, and the request is stopped at the edge before robots.txt is ever consulted. Site owners who start by editing robots.txt on the server and stop there can produce a file that reads as fully open while the domain remains closed. That inversion is the whole reason the guide exists in the shape it does.
Suganthan Mohanadasan automated the audit, and the panels changed the finding
Suganthan Mohanadasan built a free Common Crawl Visibility Checker that runs the same audit automatically, and Search Engine Journal described the result on 10 August 2026. The checker reports across four diagnostic panels. Captures per crawl shows a 12 month trend, with estimates rather than exact counts on large domains. Robots.txt history tracks stored copies backward month by month and diffs the AI crawler rules so a change is visible against a date. Block source identification matches what it finds against known templates, including Cloudflare defaults and Squarespace plugins. A live probe requests the site as CCBot/2.0 and compares that response against a normal browser and a control bot.
Search Engine Journal listed three further checks in the same tool: sitemap coverage, which diffs a submitted sitemap against the URLs actually captured; single page tracking across 12 months; and a stored copy check that compares the archived bytes with the live HTML, which is the archived version before any JavaScript execution.
The story here is not the tool. It is that automating a manual scorecard made a class of problem visible that the manual scorecard structurally could not see.
Two blocks that never appear in a robots.txt file
Search Engine Journal named two kinds of block that do not show up when you read a robots.txt by hand. The first is a CDN default. Since 1 July 2025, Cloudflare blocks AI crawlers by default on every new domain it serves, and Search Engine Journal reported that Cloudflare's managed robots.txt reached 3.8 million additional domains. The second is an edge rule: a WAF challenging CCBot while robots.txt reads clean.
Both of those defeat the check most people would run. If your test is to open the file at /robots.txt and look for a Disallow line aimed at CCBot, a clean file tells you nothing about whether the crawler was served a challenge page at the edge. Robots.txt and the edge are two independent gates, and they are allowed to disagree.
The figures below are the ones Search Engine Journal put around the scale of AI crawler blocking, and they are worth reading together rather than one at a time.
| Figure reported | What it measures |
|---|---|
| 2.14 billion pages | Size of the July 2026 Common Crawl |
| About 492,000 sites | Sites that mention CCBot in robots.txt, roughly 95% of them blocking it |
| 75% of top US and UK publishers | Share blocking AI training crawlers, per a BuzzStream measurement cited by Search Engine Journal |
| 1 July 2025 | Date from which Cloudflare blocks AI crawlers by default on every new domain it serves |
| 3.8 million additional domains | Reach of Cloudflare's managed robots.txt |
Read together, those numbers say something specific. The 492,000 figure is the count of sites that made a deliberate decision and wrote it down. The 3.8 million figure is the count of domains where a decision was made for them by a default. The second number is much larger than the first, which is why an audit that only reads robots.txt is looking in the smaller of the two places.
What CCBot not executing JavaScript does to a client-rendered site
Search Engine Journal quoted the constraint directly: CCBot does not execute JavaScript, so content that only exists after hydration leaves an empty shell in the training data. Search Engine Journal also noted that the size limit on stored pages increased in March 2025 from 1 MiB to 5 MiB.
For a React or Next.js site, that puts the question on the rendering mode rather than on the framework. A page rendered on the server, or statically generated at build time, arrives as HTML with the body copy already in it. A page that ships a near-empty div and fills it from a client-side fetch arrives as a near-empty div. The visual result in a browser is identical. The archived result is not, and the stored copy check in the automated tool exists precisely to show that gap without you having to guess.
This is also the check that is easiest to get wrong by eye. Viewing source in a browser can show a hydrated DOM depending on how you look, and a page that scores well on rendering tests aimed at Googlebot can still be an empty shell for a crawler that does no rendering at all. The comparison that matters is the raw response body against what a reader sees.
How to run the CCBot check yourself, and how to read the result
The manual version described by Search Engine Journal is a single request. You fetch your own homepage while identifying as CCBot, then fetch it again as a normal browser, then compare the two responses. Three outcomes are worth separating.
Same HTTP status, same body
The origin is serving the crawler the same page it serves a reader. This is the state the guide is steering site owners toward, and it is the point at which robots.txt and sitemap discovery start to matter.
Different status code, or a challenge page
A 403, a 503, or an interstitial challenge returned to the CCBot user agent while the browser request succeeds is the edge-rule case Search Engine Journal described. Nothing in robots.txt will explain it, because robots.txt was not the thing that stopped the request. The fix lives in the CDN or WAF configuration, which is why the guide puts CDN settings first.
Same status, but a body with no readable content
This is the client-side rendering case. The crawler was allowed in and was handed a shell. No amount of crawler permission fixes it; the rendering path has to change.
Whichever result you get, log the date you ran it. Cloudflare's default applies to domains created after 1 July 2025, so the answer for a given domain can differ from a sibling domain in the same account purely because of when it was registered. That is a configuration fact, not a performance signal, and it is the kind of thing that is much easier to explain when you have the date of the last known-good check written down. A structured technical SEO audit is the natural place to keep that record.
Presence in the training corpus is not the same as being cited
This is the limit that has to be stated plainly, because it is where most coverage of this topic goes wrong. Search Engine Journal was explicit: being in Common Crawl does not put you in a model. Every training set filters the crawl independently, so inclusion in the archive is a precondition rather than a guarantee. The article also noted that thin coverage on its own proves nothing, since Common Crawl samples the web rather than copying all of it.
There is a second distinction underneath that one. Common Crawl feeds training. It is not the retrieval layer that an AI assistant uses when it goes and fetches live pages to answer a question in the moment. Conflating the two produces bad advice in both directions: a site that is well represented in the archive can still be absent from live citations, and a site that is absent from the archive can still be cited by a system that fetches at query time. The article offers no evidence that Common Crawl presence causes AI citations, and gives no per-model breakdown of which systems used which crawl. Treat crawler accessibility as removing an obstacle, not as buying a result. If citations in AI answers are the goal, that is the separate discipline covered on the AI SEO and generative engine optimisation side of the work.
What this means for Thai marketers
The Search Engine Journal article discusses no market specifically. There are no Thailand figures in it, no Asia breakdown, and nothing about Thai-language content in the archive, so anyone claiming otherwise is adding something the source did not say.
The mechanism, though, is not regional. A Thai site sitting behind Cloudflare on a domain registered after 1 July 2025 falls inside exactly the default-block case the article describes, because the default is applied by the CDN and not by geography. That covers a lot of recently launched Thai sites, and it takes about five minutes to rule in or out. The same applies to the rendering question: a Thai-language site built as a client-rendered single page application archives as an empty shell for the same reason an English one does.
One local nuance is worth flagging honestly as reasoning rather than reporting. Thai-language content is a small share of the open web to begin with, which means the sampling caveat Search Engine Journal raised, that Common Crawl samples rather than copies, bites harder on a Thai site than on an English one. Low capture counts on a Thai domain are therefore weaker evidence of a block than the same counts on a large English domain. The live probe is the check that settles it, not the capture count.
FAQ
Is this live in Thailand, or is it a US-only thing?
It is neither announced for a market nor limited to one. Common Crawl is a global archive and the Cloudflare default described by Search Engine Journal applies to new domains that Cloudflare serves, regardless of country. The article does not discuss Thailand or any other individual market.
Do I have to do anything right now?
Only one thing is time-sensitive, and only if your domain was created after 1 July 2025 and sits behind Cloudflare. In that case the default described by Search Engine Journal applies to you until someone changes it, and a single request made as CCBot will tell you whether it does. Everything else in the guide is housekeeping that can wait for your next technical review.
Will being in Common Crawl get me cited by ChatGPT or Gemini?
No, and Search Engine Journal said so directly: being in Common Crawl does not put you in a model. Training sets filter the crawl on their own terms, and the article gives no per-model breakdown of which systems used which crawl. Common Crawl presence is about the training corpus, not about live retrieval at the moment a question is asked.
Should I block CCBot to protect my content?
The source does not make a recommendation either way, and this is a business decision rather than a technical one. What the article does report is the scale of the choice: roughly 95% of the sites that mention CCBot in robots.txt are blocking it, and 75% of top US and UK publishers block AI training crawlers according to a BuzzStream measurement it cites. A publisher licensing its archive and a services business trying to be found are not in the same position.
My robots.txt allows CCBot. Am I fine?
Not necessarily, and that is the article's central finding. A WAF can challenge CCBot at the edge while robots.txt reads clean, and a CDN default can block the crawler before robots.txt is consulted at all. The file being open is one gate. The edge is a second, independent one.
Where to take this next
The five-minute version of this is worth doing on any site you are responsible for: request the homepage as CCBot, request it as a browser, compare the status codes and the body. If they differ, the answer is in the CDN rather than in a text file. If the body comes back empty of readable content, the answer is in the rendering path. The full report is in Search Engine Journal's article. If you would rather have that checked as part of a wider look at how your site is reachable by search and AI crawlers, that is a conversation worth having with someone who does it regularly.
