Billiger.de Products: Bilingual Entity Matching Benchmark released

We are happy to announce the release of Billiger.de Products, a bilingual benchmark for evaluating entity matching systems on German, English, and cross-language product data. Entity matching is the task of determining whether two records refer to the same real-world entity, for example, whether two offers from different online shops describe the same product.

Most existing entity matching benchmarks contain records in only one language, typically English. This limits their suitability for evaluating systems in multilingual settings, such as cross-country price comparison. Billiger.de Products addresses this gap by providing aligned German and English benchmark datasets that enable controlled experiments on the influence of language on entity matching performance.

The benchmark was developed in collaboration with solute GmbH in Karlsruhe, the company operating the German price-comparison platform billiger.de. It contains 13,730 product offers describing 2,168 distinct products from thirteen product categories. In addition to the original German product offers, the benchmark provides machine-translated English versions of all records. The German and English datasets use identical splits, labels, and record identifiers, allowing systems to be evaluated on German data, English data, and mixed German–English record pairs.

Following the multidimensional design of the WDC Products benchmark, Billiger.de Products controls task difficulty along three dimensions: the size of the available training set, the proportion of difficult corner cases, and the proportion of products in the test set that are not observed during training.

We validate the benchmark using six supervised entity matching methods representing different generations of entity matching technology, as well as GPT-5.2 in a zero-shot setting. The experiments demonstrate that Billiger.de Products remains challenging for traditional feature-based methods and supervised neural matchers. While the strongest supervised methods achieve high performance under favorable conditions, their results decline considerably when the test set contains difficult product variants or products that were not included in the training data.

The cross-language experiments reveal a clear difference between supervised matching methods and the evaluated large language model. Traditional and supervised neural matchers trained on German product pairs retain much of their performance when both records are translated into English, but lose around ten F1 points when matching mixed German–English pairs. These systems appear to depend strongly on lexical overlap and other same-language signals learned from the training data. In contrast, zero-shot GPT-5.2 achieves the best results across all evaluated language combinations and shows no meaningful performance loss when matching records across languages. The results indicate that LLMs already provide strong zero-shot capabilities for multilingual entity matching, while conventional supervised methods continue to struggle with language transfer.

All benchmark datasets, cross-language test sets, baseline implementations, and experiment reproduction code are publicly available.

More information about Billiger.de Products is found:

on the project website
and in the accompanying GitHub repository

We hope that the benchmark will prove useful to the entity matching community and support the development of computationally cheap methods that can reliably match product offers in multilingual settings.

Back