Automatic sign language to text translation using MediaPipe and transformer architectures

• Tipo: Artigo em Revista Científica Internacional
• Data: 2025

Autores

Maia, WF (Autor) Outra; António Mendes Lopes (Autor) FEUP; David, SA (Autor) Outra

Revista

Título: Neurocomputing

Vol. 642

ISSN: 0925-2312

Editora: Elsevier

Indexação

ISI Web of Knowledge - 0 Citações

Scopus - 0 Citações

Outras Informações

ID Authenticus: P-018-RSC

DOI: 10.1016/j.neucom.2025.130421

Abstract (EN): This study presents a transformer-based architecture for translating Sign Language to spoken language text using embeddings of body keypoints, with the mediation of glosses. To the best of our knowledge, this work is the first to successfully leverage body keypoints for Sign Language-to-text translation, achieving comparable performance to baseline models without reducing translation quality. Our approach introduces extensive augmentation techniques for body keypoints, and convolutional keypoint embeddings, and integrates Connectionist Temporal Classification Loss and position encoding for Sign2Gloss translation. For the Gloss2Text stage, we employ fine-tuning of BART, a state-of-the-art transformer model. Evaluation on the Phoenix14T dataset demonstrates that our integrated Sign2Gloss2Text model achieves competitive performance, with BLEU-4 scores that show marginal differences compared to baseline models using pixel embeddings. On the How2Sign dataset, which lacks gloss annotations, direct Sign2Text translation posed challenges, as reflected in lower BLEU-4 scores, highlighting the limitations of gloss-free approaches. This work addresses the narrow domain of the datasets and the unidirectional nature of the translation process while demonstrating the potential of body keypoints for Sign Language Translation. Future work will focus on enhancing the model's ability to capture nuanced and complex contexts, thereby advancing accessibility and assistive technologies for bridging communication between individuals with hearing impairments and the hearing community.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Nº de páginas: 13