TL;DR
- Cloudflare and OpenAI said on 15 October 2026 that their discovery pilot, begun in July 2026, helped OpenAI index hundreds of millions of new pages.
- The pilot used a participating cohort of about 1% of Cloudflare's network; Cloudflare supplies signals and OpenAI independently evaluates and ranks pages.
- Signals filter nonexistent URLs, catch-all homepage redirects and duplicates, flag when a page is worth revisiting, and in tests flag single compromised pages.
- The post says a signal does not grant permission to access a page and the work uses metadata, not page content for model training.
Cloudflare and OpenAI said on 15 October 2026 that a joint research pilot using signals from Cloudflare's network has helped OpenAI's search crawler discover and index hundreds of millions of new web pages for ChatGPT Search. In a post on the Cloudflare blog, co-written by Cloudflare's Matthew Conroy and Jack Galilee and OpenAI's Henry Aspegren and James Balfour, the companies say the pilot began in July 2026 and used a participating cohort of about 1% of Cloudflare's total network.
The post is short on per-site detail, but it is specific about the mechanism. Cloudflare supplies discovery signals; OpenAI decides what to index and how to rank it. For any business that wants to be cited in ChatGPT answers, the first step is being in OpenAI's search index, and this pilot changes how some pages get there.
What Cloudflare and OpenAI announced on 15 October 2026
According to the post, the two companies launched the research pilot in July 2026 to find out "whether signals from Cloudflare's network could help ChatGPT users discover new content more efficiently". Cloudflare first announced the pilot in a press release dated 8 July 2026, which named content freshness, traffic quality and actual page changes as the kinds of network insights it would test. The early results, published on 15 October, are summed up in one claim: the pilot "has enabled OpenAI to discover and index hundreds of millions of new web pages". The post describes these as "previously unknown pages" now in OpenAI's search index.
The division of labour is stated plainly. Cloudflare's network "provides signals that can help OpenAI's search crawler discover more useful pages", and OpenAI "then independently evaluates and ranks how those pages appear based on how relevant and helpful they are". In other words, the signals help a page get found. They do not decide whether it is shown or cited.
The authors also connect the work to OpenAI's product direction, saying that with GPT-6 and Intelligent UI, OpenAI is making ChatGPT "an even better place to shop, learn, and find answers", and that this discovery work complements those changes by helping ChatGPT find more useful, up-to-date information.
Why discovery is the problem the pilot targets
The post argues that the most useful pages are often the least linked. Its examples are the manual for an exact washing machine model, current passport requirements and fees, and a local restaurant's latest menu and opening hours. "These pages can be valuable even if few other sites link to them," the authors write.
The post does not list the exact signals Cloudflare shares, but it says the research "uses metadata about the web for search and discovery, rather than sharing page content for model training".
How the pilot works: four mechanisms described in the post
The post describes what Cloudflare and OpenAI learned about separating real pages from noise before a crawler spends effort on them. The table below summarises each mechanism as the post describes it.
| Mechanism | What the 15 October post says |
|---|---|
| Filtering URLs that do not exist | Bots repeatedly request files that are not there, which can make a nonexistent page "look interesting"; recognising this cuts wasted work before a fetch. |
| Redirect and duplicate detection | Some sites redirect almost any address to the homepage, and other addresses lead to copies of the same content; both are filtered out. |
| Change signals | Signals tell the crawler when a page is worth another visit and when it is not, to avoid repeatedly downloading unchanged pages. |
| Harmful-page flagging (in testing) | Signals that flag a single compromised page, such as a planted fake login form, without penalising the whole domain. |
On change signals, the post makes a cost argument: "For a website owner, a crawler that repeatedly downloads an unchanged page adds load without bringing any benefit." Filtering keeps crawlers away from pages that do not exist or redirect elsewhere, and change signals tell them when to come back.
On harmful pages, the authors say "early results suggest these signals can help identify pages that should be kept out of an index". The example is a legitimate site with one page compromised by an attacker. Flagging only that page "keeps people safe without penalizing the site owner".
The post also admits a trade-off. "Finding new pages can initially increase crawling as those pages are evaluated and indexed." Site owners in the cohort may have seen more crawl activity at first, not less. The stated goal is to keep discovering and stay current while reducing the work involved.
What the pilot does not change: access and permission
The post is explicit that discovery is not permission. "A signal about a page doesn't grant permission to access it," the authors write, and "domain owners' choices matter throughout". It points site owners to Cloudflare's AI Crawl Control to understand and manage how AI systems access their sites.
That makes the existing access controls the first thing to check. OpenAI's own crawler documentation describes separate user agents for separate jobs. OAI-SearchBot is used to surface websites in ChatGPT's search features, and OpenAI says sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot is used to crawl content that may be used to train OpenAI's foundation models, and disallowing it is the way to say content should not be used for training. ChatGPT-User handles certain user-initiated actions and, according to OpenAI, is not used to decide whether content appears in search. OpenAI also says it can take about 24 hours for its search systems to adjust after a robots.txt change.
Cloudflare's documentation describes AI Crawl Control, formerly AI Audit, as available on all plans. It shows which AI services access a site, lets owners set allow or block rules per crawler, and tracks whether crawlers follow robots.txt directives.
Our reading of the post together with OpenAI's documentation: a site that blocks OAI-SearchBot, whether in robots.txt or through a Cloudflare rule, is not made visible to ChatGPT Search by this pilot. For a site in the cohort that allows it, the signals may help more of its pages get found, including pages that few others link to.
Who is affected, and who is not
Only sites in the participating cohort, about 1% of Cloudflare's network, were part of the pilot. The post does not say how those sites were chosen, whether site owners opted in, which countries or languages they covered, or whether the pilot will be widened to the rest of the network. Sites not on Cloudflare were outside it.
The "hundreds of millions" figure is an aggregate across the cohort. The post gives no per-site numbers, no traffic or referral data from ChatGPT, and no measure of how many of the newly indexed pages were later cited in answers. The authors say the next phase will "measure what happens after content is discovered" and that they are working with OpenAI "to understand how better site discovery can translate into real business outcomes". Those outcomes have not been published.
What this means for AI search visibility
This is analysis, not a claim from the post. In AI search, citation depends on several things happening in order: the page has to be discovered, fetched, indexed, judged relevant and then chosen as a source. Most advice on generative engine optimisation focuses on the later stages, such as clear answers, structured content and authority. The Cloudflare and OpenAI discovery pilot is about the first stage. A page that OpenAI's search index never learns about cannot be cited, however good it is.
The mechanisms in the post also describe what a search crawler treats as waste: URLs that do not exist, catch-all redirects to the homepage, duplicate copies, and unchanged pages fetched again and again. Sites that make those patterns common are making their own useful pages harder to separate from noise. That is consistent with standard technical SEO practice, and it now applies to AI search crawlers as well as to Google.
What to check on your own site
The checklist below is practical analysis based on the post and on OpenAI's and Cloudflare's documentation. The post itself does not give site owners instructions beyond pointing to AI Crawl Control.
- Read your robots.txt for OAI-SearchBot and GPTBot. Decide separately whether you want to appear in ChatGPT search and whether you allow training crawls. They are independent settings.
- If you use Cloudflare, open AI Crawl Control. Check which AI crawlers are allowed or blocked and whether a block rule is stopping OAI-SearchBot when you meant to stop only training crawlers.
- Return real 404 or 410 responses for pages that do not exist. The post names sites that redirect almost any address to the homepage as a pattern crawlers learn to filter.
- Consolidate duplicates. Parameter variants and copied pages are the "copies of the same content" the post says get filtered before a fetch.
- Make sure low-link pages are reachable. Pricing pages, branch pages, product manuals and Thai-language service pages are the kind of specific pages the post says are valuable even with few links. They should not sit behind login walls or noindex rules by accident.
- Update pages when facts change. Change signals reward pages that actually change. The post does not say anything about sitemaps or lastmod dates, so do not assume that editing a date alone triggers a revisit.
- Watch for compromised pages. The harmful-page signals are still in testing, but a planted login form on one URL is exactly the case the post describes.
Relevant Audience's ChatGPT SEO service covers crawler access, indexing and citation readiness for OpenAI's search, and a technical SEO audit finds the redirect, duplicate and soft-404 patterns described above.
What the post did not say
- It did not list the specific signals Cloudflare shares, beyond saying they are metadata.
- It did not say how the roughly 1% cohort was selected, whether owners consented individually, or when the pilot might expand.
- It gave no figures for traffic, citations or referrals from ChatGPT to the newly indexed pages.
- It did not give a launch date for the harmful-page signals, which are still described as being tested.
- It did not say whether any money changes hands between the companies, or whether the arrangement is exclusive.
What this means for Thai marketers
This section is our analysis. The post gives no country or language breakdown, so it is not possible to say whether Thai sites were in the cohort. What applies in Thailand is the same as anywhere: if your site runs on Cloudflare, your AI crawler settings there decide whether OpenAI's search crawler can reach you, so check them rather than assume. Specific pages with few links, such as branch locations, Thai-language price lists and service details, match the kind of page the post describes as worth discovering, so keep them crawlable, free of catch-all redirects, and current. If you are unsure whether ChatGPT can see your site at all, start with robots.txt and the AI Crawl Control dashboard before working on content.
FAQ
What did Cloudflare and OpenAI announce?
Cloudflare and OpenAI said on 15 October 2026 that a research pilot using Cloudflare network signals helped OpenAI discover and index hundreds of millions of new web pages for ChatGPT Search. The pilot started in July 2026 and used a cohort of about 1% of Cloudflare's network.
Does the pilot give OpenAI permission to crawl my site?
No, the post says "a signal about a page doesn't grant permission to access it", and site owners' choices still apply. According to OpenAI's crawler documentation and Cloudflare's AI Crawl Control documentation, blocking OAI-SearchBot in robots.txt or through Cloudflare's AI Crawl Control still controls whether OpenAI's search crawler can access your pages.
Is my content used to train OpenAI models through this pilot?
No, according to the post, the research uses metadata about the web for search and discovery rather than sharing page content for model training. OpenAI's crawler documentation says training crawls are governed separately by its GPTBot user agent.
Is my site part of the pilot?
The post does not say which sites were included, only that the participating cohort was about 1% of Cloudflare's total network. It gives no list, selection method or country breakdown.
Will this get my pages cited in ChatGPT?
Not by itself, because the signals only help pages get discovered and OpenAI independently evaluates and ranks them. Whether a page is cited still depends on how relevant and helpful OpenAI judges it to be.
The pilot is early and Cloudflare and OpenAI have not yet published business results. Until they do, the step site owners control is crawler access to the pages they want found.
