Deep learning-based human action recognition to leverage context awareness in collaborative assembly

• Type: Article in International Scientific Journal
• Date: 2023

Authors

Moutinho, D (Author) Other; Rocha, LF (Author) FEUP; Costa, CM (Author) Other; Luis F Teixeira (Author) FEUP; Germano Veiga (Author) Other

Journal

Title: Robotics and Computer-Integrated Manufacturing

Vol. 80

ISSN: 0736-5845

Publisher: Elsevier

Indexing

ISI Web of Knowledge - 16 Citations

Scopus - 21 Citations

Other information

Authenticus ID: P-00X-A8M

DOI: 10.1016/j.rcim.2022.102449

Abstract (EN): Human-Robot Collaboration is a critical component of Industry 4.0, contributing to a transition towards more flexible production systems that are quickly adjustable to changing production requirements. This paper aims to increase the natural collaboration level of a robotic engine assembly station by proposing a cognitive system powered by computer vision and deep learning to interpret implicit communication cues of the operator. The proposed system, which is based on a residual convolutional neural network with 34 layers and a long -short term memory recurrent neural network (ResNet-34 + LSTM), obtains assembly context through action recognition of the tasks performed by the operator. The assembly context was then integrated in a collaborative assembly plan capable of autonomously commanding the robot tasks. The proposed model showed a great performance, achieving an accuracy of 96.65% and a temporal mean intersection over union (mIoU) of 94.11% for the action recognition of the considered assembly. Moreover, a task-oriented evaluation showed that the proposed cognitive system was able to leverage the performed human action recognition to command the adequate robot actions with near-perfect accuracy. As such, the proposed system was considered as successful at increasing the natural collaboration level of the considered assembly station.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 15