Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

• Tipo: Artigo em Livro de Atas de Conferência Internacional
• Data: 2024

Autores

Jesus, Gd (Autor) Outra; Sérgio Nunes (Autor) FEUP

Ata de Conferência Internacional

Título: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Páginas: 4368-4380

Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Torino, 2024

Indexação

Scopus - 3 Citações

Outras Informações

ID Authenticus: P-010-MB8

Abstract (EN): This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an open-source web crawler and data extraction framework, and incorporates language processing components such as a tokenizer and a language identification model. The pipeline efficacy is demonstrated through successful testing with Tetun, one of Timor-Leste's official languages, resulting in the construction of a high-quality Tetun text corpus comprising 321.7k sentences extracted from over 22k web pages. The contributions of this paper include the development of a Tetun tokenizer, a Tetun language identification model, and a Tetun text corpus, marking an important milestone in Tetun text information retrieval.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Nº de páginas: 12