BATS-PT: assessing portuguese masked language models in lexico-semantic analogy solving and relation completion

• Type: Article in International Conference Proceedings Book
• Date: 2024

Authors

Oliveira, Hugo Gonçalo (Author) Other; Rodrigues, Ricardo (Author) Other; Ferreira, Bruno (Author) Other; Silvano, Maria da Purificação (Author) FLUP; Carvalho, Sara (Author) Other

Conference proceedings International

Title: Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, 12-15 March, 2024

16th International Conference on Computational Processing of Portuguese, PROPOR 2024

Santiago de Compostela, 2024

Other information

Abstract (EN): This paper presents BATS-PT, the manual translation of the lexicographic portion of the Bigger Analogy Test Set (BATS) to European Portuguese. BATS-PT covers ten types of lexicosemantic analogies and can be used for assessing word embeddings and language models. Following this, the dataset is showcased while assessing two pretrained language models for Portuguese, BERTimbau and Albertina, in two tasks: analogy solving and relation completion, both in zero- and few-shot mask-prediction approaches. Experiments reveal different performance across relations and, in both tasks, the best overall performance was achieved with BERTimbau, in a five-shot scenario. We further discuss the limitations of the reported experiments and directions towards future improvements in these tasks

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 11