The News, Explained
Cloudflare and OpenAI published early results from their joint web-discovery research on October 15. The pilot began in July 2026 and used a participating cohort representing about 1% of Cloudflare’s total network. The companies say it enabled OpenAI to discover and index hundreds of millions of new web pages. Those figures describe the participating scope and the companies’ early results. Source
Finding an address is the beginning
A search system first needs to know a page’s address before it can examine the information there. In this pilot, signals from Cloudflare’s network help OpenAI’s search crawler find potentially useful pages. A crawler is an automated program that visits web pages; a search index organizes page information so a system can retrieve it later. Source
Further decisions follow discovery. A page must be accessible, offer something distinct and help with a particular question. The announcement says OpenAI independently evaluates pages and ranks how they appear. Separating the discovery signal from the later ranking decision gives website owners a more precise way to understand what the experiment offers. Source
The announcement uses a manual for an exact washing-machine model and current passport requirements as examples of information people may need. A page can contain useful specifics even when few other sites link to it. These examples explain the research’s purpose; they are not measurements of increased visits to individual websites. Source
Signals for avoiding wasted visits
Repeated automated requests do not establish that a useful page exists at an address. The research considers patterns such as requests for missing files, addresses that redirect elsewhere and multiple copies of the same content. It is also testing signals that flag potentially harmful pages. For example, a legitimate site might contain a fake login page inserted by an attacker, making page-level identification useful. Source
Change signals are intended to reduce repeated downloads of content that has stayed the same. The companies also acknowledge that finding previously unknown pages can initially increase crawling while those pages are evaluated and indexed. The aim is to keep discovering current information while reducing unnecessary collection work. Source
Metadata does not grant access
The companies describe the research as using web metadata for search and discovery, rather than sharing page content for model training. Metadata here means signals and information about pages, distinct from the page content itself. They also state that a signal about a page does not grant permission to access it. Domain owners’ choices continue to matter, with Cloudflare’s AI Crawl Control presented as a way to understand and manage AI access. Source
The next phase is intended to improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered. The relationship between discovery and real business outcomes remains a subject for further research. The number of newly indexed pages and evidence of visits or sales therefore belong to different stages of evaluation. Source
OYOPICK’s Take
Broader discovery is promising when it gives the specific explanations and experience on smaller websites a better chance of reaching people through AI search. For publishers, however, the useful evidence goes beyond the number of addresses found. Knowing which pages get cited, whether people visit them and what collection load accompanies that activity would make participation easier to assess.
We would judge the next phase by the usefulness of the connections it creates. We hope the research preserves owners’ access choices while making post-discovery use and costs easier to evaluate. Continuing to distinguish being discovered from granting permission to use content would help both readers and publishers understand how AI search is changing.