Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

• Type: Article in International Conference Proceedings Book
• Date: 2024

Authors

de Jesus G. (Author) Other; Sérgio Nunes (Author) FEUP

Conference proceedings International

Title: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Pages: 4368-4380

Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Torino, 2024

Indexing

Scopus - 1 Citation

Other information

Authenticus ID: P-010-MB8

Abstract (EN): This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an open-source web crawler and data extraction framework, and incorporates language processing components such as a tokenizer and a language identification model. The pipeline efficacy is demonstrated through successful testing with Tetun, one of Timor-Leste's official languages, resulting in the construction of a high-quality Tetun text corpus comprising 321.7k sentences extracted from over 22k web pages. The contributions of this paper include the development of a Tetun tokenizer, a Tetun language identification model, and a Tetun text corpus, marking an important milestone in Tetun text information retrieval.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 12