Evaluating WMT 2025 Metrics shared task submissions on the SSA-MTE African challenge set

• Tipo: Artigo em Livro de Atas de Conferência Internacional
• Data: 2025

Autores

Li, Senyu (Autor) Outra; Ali, Felermino D. M. A. (Autor) Outra; Wang, Jiayi (Autor) Outra; Sousa-Silva, Rui (Autor) FLUP; Cardoso, Henrique Lopes (Autor) FEUP; Stenetorp, Pontus (Autor) Outra; Cherry, Colin (Autor) Outra; Adelani, David Ifeoluwa (Autor) Outra

Ata de Conferência Internacional

Título: Proceedings of the Tenth Conference on Machine Translation

Páginas: 913-919

10th Conference on Machine Translation

Suzhou, China, 2025

Indexação

Crossref

Outras Informações

DOI: 10.18653/v1/2025.wmt-1.65

Abstract (EN): This paper presents the evaluation of submissions to the WMT 2025 Metrics Shared Task on the SSA-MTE challenge set, a large-scale benchmark for machine translation evaluation (MTE) in Sub-Saharan African languages. The SSA-MTE test sets contains over 12,768 human-annotated adequacy scores across 11 language pairs sourced from English, French, and Portuguese, spanning 6 commercial and open-source MT systems. Results show that correlations with human judgments remain generally low, with most systems falling below the 0.4 Spearman threshold for medium-level agreement. Performance varies widely across language pairs, with most correlations under 0.4; in some extremely low-resource cases, such as Portuguese–Emakhuwa, correlations drop to around 0.1, underscoring the difficulty of evaluating MT for very low-resource African languages. These findings highlight the urgent need for more research on robust, generalizable MT evaluation methods tailored for African languages.

Idioma: Inglês

Tipo (Avaliação Docente): Científica