On the role of multimodal learning in the recognition of sign language

• Tipo: Artigo em Revista Científica Internacional
• Data: 2019

Autores

Pedro M. Ferreira (Autor) Outra; Jaime S. Cardoso (Autor) FEUP; Ana Rebelo (Autor) Outra

Revista

Título: Multimedia Tools and Applications

Vol. 78 Nº 8

Páginas: 10035-10056

ISSN: 1380-7501

Editora: Springer Nature

Indexação

ISI Web of Knowledge - 2 Citações

Scopus - 6 Citações

INSPEC

Outras Informações

ID Authenticus: P-00P-M59

DOI: 10.1007/s11042-018-6565-5

Abstract (EN): Sign Language Recognition (SLR) has become one of the most important research areas in the field of human computer interaction. SLR systems are meant to automatically translate sign language into text or speech, in order to reduce the communicational gap between deaf and hearing people. The aim of this paper is to exploit multimodal learning techniques for an accurate SLR, making use of data provided by Kinect and Leap Motion. In this regard, single-modality approaches as well as different multimodal methods, mainly based on convolutional neural networks, are proposed. Our main contribution is a novel multimodal end-to-end neural network that explicitly models private feature representations that are specific to each modality and shared feature representations that are similar between modalities. By imposing such regularization in the learning process, the underlying idea is to increase the discriminative ability of the learned features and, hence, improve the generalization capability of the model. Experimental results demonstrate that multimodal learning yields an overall improvement in the sign recognition performance. In particular, the novel neural network architecture outperforms the current state-of-the-art methods for the SLR task.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Nº de páginas: 22