Evaluating WMT 2025 Metrics shared task submissions on the SSA-MTE African challenge set

• Type: Article in International Conference Proceedings Book
• Date: 2025

Authors

Li, Senyu (Author) Other; Ali, Felermino D. M. A. (Author) Other; Wang, Jiayi (Author) Other; Sousa-Silva, Rui (Author) FLUP; Cardoso, Henrique Lopes (Author) FEUP; Stenetorp, Pontus (Author) Other; Cherry, Colin (Author) Other; Adelani, David Ifeoluwa (Author) Other

Conference proceedings International

Title: Proceedings of the Tenth Conference on Machine Translation

Pages: 913-919

10th Conference on Machine Translation

Suzhou, China, 2025

Indexing

Crossref

Other information

DOI: 10.18653/v1/2025.wmt-1.65

Abstract (EN): This paper presents the evaluation of submissions to the WMT 2025 Metrics Shared Task on the SSA-MTE challenge set, a large-scale benchmark for machine translation evaluation (MTE) in Sub-Saharan African languages. The SSA-MTE test sets contains over 12,768 human-annotated adequacy scores across 11 language pairs sourced from English, French, and Portuguese, spanning 6 commercial and open-source MT systems. Results show that correlations with human judgments remain generally low, with most systems falling below the 0.4 Spearman threshold for medium-level agreement. Performance varies widely across language pairs, with most correlations under 0.4; in some extremely low-resource cases, such as Portuguese–Emakhuwa, correlations drop to around 0.1, underscoring the difficulty of evaluating MT for very low-resource African languages. These findings highlight the urgent need for more research on robust, generalizable MT evaluation methods tailored for African languages.

Language: English

Type (Professor's evaluation): Scientific