Cloudflare says pilot added hundreds of millions of pages to OpenAI's index

Cloudflare says its research pilot with OpenAI has enabled OpenAI to discover and index hundreds of millions of new web pages. The companies launched the pilot in July; the early results cover a participating cohort of approximately 1% of Cloudflare's total network. Cloudflare supplies network signals that help identify useful pages and filter nonexistent addresses, redirects and duplicate content before a crawler fetches them. OpenAI independently evaluates and ranks the pages. The next phase will test better page selection and change signals and measure outcomes after discovery. The research uses web metadata for search and discovery rather than sharing page content for model training. A discovery signal does not grant permission to access a page.