SSA-COMET: Do LLMs outperform learned metrics in evaluating MT for under-resourced African languages?

• Tipo: Artigo em Livro de Atas de Conferência Internacional
• Data: 2025

Autores

Li, Senyu (Autor) Outra; Wang, Jiayi (Autor) Outra; Ali, Felermino D. M. A. (Autor) Outra; Cherry, Colin (Autor) Outra; Deutsch, Daniel (Autor) Outra; Briakou, Eleftheria (Autor) Outra; Sousa-Silva, Rui (Autor) FLUP; Cardoso, Henrique Lopes (Autor) FEUP; Stenetorp, Pontus (Autor) Outra; Adelani, David Ifeoluwa (Autor) Outra

Ata de Conferência Internacional

Título: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

Páginas: 12991-13010

2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

Suzhou, China, 2025

Indexação

Crossref

Outras Informações

DOI: 10.18653/v1/2025.emnlp-main.656

Abstract (EN): Evaluating machine translation (MT) quality for under-resourced African languages remains a significant challenge, as existing metrics often suffer from limited language coverage and poor performance in low-resource settings. While recent efforts, such as AfriCOMET, have addressed some of the issues, they are still constrained by small evaluation sets, a lack of publicly available training data tailored to African languages, and inconsistent performance in extremely low-resource scenarios. In this work, we introduce SSA-MTE, a large-scale human-annotated MT evaluation (MTE) dataset covering 14 African language pairs from the News domain, with over 73,000 sentence-level annotations from a diverse set of MT systems. Based on this data, we develop SSA-COMET and SSA-COMET-QE, improved reference-based and reference-free evaluation metrics. We also benchmark prompting-based approaches using state-of-the-art LLMs like GPT-4o, Claude-3.7 and Gemini 2.5 Pro. Our experimental results show that SSA-COMET models significantly outperform AfriCOMET and are competitive with the strongest LLM Gemini 2.5 Pro evaluated in our study, particularly on low-resource languages such as Twi, Luo, and Yoruba. All resources are released under open licenses to support future research.

Idioma: Inglês

Tipo (Avaliação Docente): Científica