Like most people, I search on Google or on Bing. I type a few words, and the answer arrives in a fraction of a second. That speed comes from work done long before my query: robots have crawled the Web, kept the text of the pages and filed that text in a giant catalogue.

This catalogue is called an index, and it is very expensive to build. That is also why web search is played out between a handful of companies, almost all of them American. Anyone who wants to launch an engine today has to either redo that collection, or rent access to somebody else’s.

Two European names answer that situation, and they mean two different things. Open Web Search is a project funded by the European Union, which built this machinery in common. Open Web Index is the catalogue that this machinery produces and publishes, open to whoever wants to use it.

This article explains what the project does, what its index holds, how that index is built, how to reach it and what its licence allows. The examples are drawn from French data.

Telling the index apart from the search engine

An engine’s index has a precise name, the inverted index. It stores the text of pages the other way round from what one imagines: instead of starting from a page to find its words, it starts from a word to find its pages.

Under the entry “software”, it gives the list of pages where that word appears. Searching for “free software” then comes down to crossing two of these lists, and sorting what remains. This structure explains the speed of the answer, and it explains the cost too, because the whole Web has to be read to fill it.

An engine adds a second layer on top. It ranks the answers, discards the pages judged useless, applies editorial rules and draws the interface. The first layer is expensive and looks much the same from one engine to the next. The second is what tells two services apart.

Two boxes linked by an arrow: on the left the shared foundation that crawls the pages, extracts their text and builds the index, on the right the engine layer that ranks the answers, filters and draws the interface

The Open Web Index publishes the foundation. The engine layer remains the work of each service.

This split explains the idea behind the project. A single collection can feed many engines, provided its result is published in a format that others know how to read.

Placing the Open Web Search project

Building this foundation takes machines, time and money. In Europe, that spending took the form of a public research project.

The OpenWebSearch.eu project is funded by the Horizon Europe programme. This programme is the main instrument through which the European Union funds research and innovation, endowed with 93.5 billion euros for the 2021 to 2027 period. It hands out that money through calls for proposals, to consortia that bring together laboratories from several countries.

The project started on 1 September 2022 and ended on 28 February 2026, on a budget of 8,502,621.75 euros fully covered by the European Union. It brings together fourteen research and computing centres spread over seven countries, among them CERN, the Bavarian Academy of Sciences, Radboud University and the Open Search Foundation, a German non-profit association. Coordination falls to the University of Passau.

The stated goal is explicit in the official title of the project: supporting the digital sovereignty of Europe. The project website presents access to information as controlled today by a small number of non-European players, and offers a shared technical foundation rather than one more engine.

The end of the funding leaves the sequel open. The project describes a sustainability phase, with the idea of an index maintained as public infrastructure and the choice of a suitable legal form, for example a European non-profit entity. The infrastructure keeps publishing daily datasets in August 2026.

Looking at what the Open Web Index holds

The Open Web Index portal publishes the state of the infrastructure and the figures of the crawl. This page can be read without an account and updates every day.

Portal dashboard: two computing centres online, 1,662.06 TB crawled, 328 WARC datasets, 1,963 public datasets, 35.09 TB of index, and a language breakdown dominated by English Portal dashboard: two computing centres online, 1,662.06 TB crawled, 328 WARC datasets, 1,963 public datasets, 35.09 TB of index, and a language breakdown dominated by English

The Open Web Index portal on 16 August 2026.

Two volumes stand apart. The 1,662.06 terabytes of crawled data correspond to the pages archived as they were received. The 35.09 terabytes of the Open Web Index correspond to the processed result, far more compact, which gathers the text of the pages, their metadata and the indexes. The project website also announces 9.14 billion addresses and 185 languages.

The breakdown by language deserves a careful look. English accounts for 38% of the pages, German for 8%, French, Spanish and Chinese for 6% each. A French corpus does exist, while staying a minority inside a set dominated by English.

Two computing centres feed the whole thing, IT4I in Czechia and LRZ in Germany. The dashboard shows their state, the number of active processing pipelines and the compute hours consumed, 51,600 core hours at the time of the capture.

Following how the index is built

Building it starts with a crawler, a program that follows links from page to page and records what it receives. The one of the project is called OWLer. It obeys the robots.txt file through which a site declares what it accepts to have collected, and it writes its responses in the WARC format, the standard archive of web crawls.

These archives then go through a processing chain. The Resiliparse library, running on Apache Spark, separates the main content of a page from its menus, its banners and its adverts. A last step builds the inverted index in the CIFF format, for Common Index File Format, and files the metadata in Parquet files, a compressed tabular format designed for analysis.

Production chain of the Open Web Index: OWLer collects the pages, Resiliparse extracts their content, an indexer produces the Parquet and CIFF files, then the datasets are published and downloaded onto a workstation

The chain stops at the publication of the datasets. What comes next belongs to each application.

This choice of formats matters more than it seems. CIFF was designed so that different engines read the same index. The status page of the project lists Apache Lucene, Elasticsearch, OpenSearch and PyTerrier as import targets, which avoids forcing a single engine on those who reuse the data.

Opening the catalogue of datasets

The index is published as datasets. A dataset is a slice of time, most often a single day, for a given computing centre. The portal presents them as cards, with their size, their number of files and their collection.

Three cards of published datasets on the portal, carrying the GPU and MAIN labels, their size in gigabytes, their number of files and the matching owilix remote pull command Three cards of published datasets on the portal, carrying the GPU and MAIN labels, their size in gigabytes, their number of files and the matching owilix remote pull command

Each dataset card shows its collection, its volume and the download command.

The collections separate sets of a different nature. The main collection gathers the general crawl. Others group legal pages, the Curlie directory, Wikipedia articles or national corpora. Each card also states its licence and the exact command that fetches the dataset.

Downloading requires an account. A visitor freely reads the statistics and the catalogue, then has to sign in to obtain the files. Open here means that the data is published and reusable, and not that reaching it skips any registration.

Querying the index from a workstation

The project provides a command line client, OWILIX, whose documentation describes the installation, the remote search and the download. Installing it creates an isolated Python environment and adds the command to the system path:

curl -fsSL https://openwebindex.net/owilix/install.sh | sh

A remote search queries the shared index without downloading anything. I ran this one on French, asking for the first five answers:

owilix remote search "logiciel libre" --language fra --limit 5

The command looks for the terms in the main content of the pages and returns a list of addresses, with a score and the crawl date.

Output of the OWILIX command for the query logiciel libre: five addresses ranked with their score and their crawl date

The remote search serves to probe the index before choosing datasets to download.

This first query took 34.7 seconds, and my other attempts between 29 and 56 seconds. These durations place the use of the tool: it explores a shared index, in the manner of an analytical query, and not at the speed expected from a consumer engine.

Downloading follows the same logic. To build myself a working corpus, I fetched 85 French datasets, that is 721 files and 69.2 compressed gigabytes, holding 5,450,238 pages. A count by domain extension finds 411 different suffixes there, with 1,970,732 pages in .fr and 1,754,029 in .com, followed by Switzerland, Belgium and Canada. The language filter therefore catches far more than the national domain.

Seeing an engine built on the index

The portal gives access to OURRS, for Open Universal Research and Retrieval System, an experimental search engine operated by the Data Science chair of the University of Passau. It uses the Open Web Index and can be queried without an account, which gives a concrete idea of what the foundation allows.

Interface of the OURRS engine for the query logiciel libre: about 191 results returned in less than a second, with pages from gnu.org and from the Agenda du Libre Interface of the OURRS engine for the query logiciel libre: about 191 results returned in less than a second, with pages from gnu.org and from the Agenda du Libre

The query for free software in OURRS, an engine built on the Open Web Index.

The query “logiciel libre”, French for free software, returns about 191 documents in a fraction of a second, with pages from gnu.org, from the Agenda du Libre and from the French committee for open science. The engine announces coverage of 891.3 million pages, 650.5 million entities and 29.5 million sites, a share of the index rather than all of it.

This service remains a research prototype, and it says so. Its operators warn that the data may be reset while the index is being rebuilt. Some extracts still show badly decoded accented characters, a processing flaw visible to the naked eye.

Knowing what the licence allows

Access to the files is governed by the Open Web Index Licence 1.0, accepted at the first sign-in. It reserves the index for personal, non-commercial and research use, which rules out a commercial service built on it for now.

It also imposes a precise notice. Any application or publication that uses a partition of the index must state that the work relies on files produced by the OpenWebSearch.eu project, funded by Horizon Europe under grant agreement No 101070014.

The rest of the text recalls the experimental status of the service. It is provided as is, without any guarantee of availability or accuracy, the licensor may revoke access and ask for the data to be destroyed, and German law applies in case of dispute.

Limits

The index remains a pilot. The portal presents it in beta, its services may change or become unavailable, and its coverage of 35.09 terabytes stays partial at the scale of the Web.

The daily datasets overlap. The same address comes back at several dates, with contents that are sometimes identical. A serious corpus therefore has to decide whether it keeps the full history or only the most recent version of each page.

The enrichment fields are unevenly filled. Across the 5,450,238 pages of my French corpus, the quality score is empty everywhere, and the topical labels from Curlie cover 1,229,959 pages, that is 22.6%. These fields can complete a ranking, provided they are not its only basis.

The institutional future of the project remains to be settled. European funding stopped in February 2026, the final legal form of the infrastructure is still under study, and the current licence closes the door on the commercial uses that would secure other revenue.

Conclusion

Open Web Search is the project, the Open Web Index is its result. The first pooled the most expensive part of a search engine, the second publishes it as daily datasets of pages, of metadata and of indexes.

The way I search day to day stays the same, and the difference lies elsewhere. The most expensive brick of an engine now exists in Europe, and it can be downloaded. OURRS shows that an engine can be built on it, OWILIX makes it possible to fetch and analyse the datasets from a workstation, and the ranking, the scope and the interface remain the work of each application. That is precisely the part the project chose to leave open, and the one where I intend to spend time.