Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*

• Type: Article in International Conference Proceedings Book
• Date: 2023

Authors

Rodrigues, J (Author) Other; Gomes, L (Author) Other; Silva, J (Author) Other; Branco, A (Author) Other; Santos, R (Author) Other; Henrique Lopes Cardoso (Author) FEUP; Osório, T (Author) Other

Conference proceedings International

Title: PROGRESS IN ARTIFICIAL INTELLIGENCE, EPIA 2023, PT I

Pages: 441-453

22nd EPIA Conference on Artificial Intelligence (EPIA)

Azores, PORTUGAL, SEP 05-08, 2023

Indexing

ISI Web of Knowledge - 0 Citations

Scopus - 0 Citations

Other information

Authenticus ID: P-00Z-KWK

DOI: 10.1007/978-3-031-49008-8_35

Abstract (EN): To advance the neural encoding of Portuguese (PT), and a fortiori the technological preparation of this language for the digital age, we developed a Transformer-based foundation model that sets a new state of the art in this respect for two of its variants, namely European Portuguese from Portugal (PT-PT) and American Portuguese from Brazil (PT-BR). To develop this encoder, which we named Albertina PT-*, a strong model was used as a starting point, DeBERTa, and its pre-training was done over data sets of Portuguese, namely over a data set we gathered for PT-PT and over the brWaC corpus for PT-BR. The performance of Albertina and competing models was assessed by evaluating them on prominent downstream language processing tasks adapted for Portuguese. Both Albertina versions are distributed free of charge and under a most permissive license possible and can be run on consumer-grade hardware, thus seeking to contribute to the advancement of research and innovation in language technology for Portuguese.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 13