Detecting Concepts and Generating Captions from Medical Images: Contributions of the VCMI Team to ImageCLEFmedical Caption 2023

• Tipo: Artigo em Livro de Atas de Conferência Internacional
• Data: 2023

Autores

Torto, IR (Autor) Outra; Patrício, C (Autor) Outra; Montenegro, H (Autor) Outra; Gonçalves, T (Autor) Outra; Jaime S Cardoso (Autor) FEUP

Ata de Conferência Internacional

Título: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023.

Páginas: 1653-1667

24th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF-WN 2023

Thessaloniki, 18 September 2023 through 21 September 2023

Indexação

Scopus - 1 Citação

Outras Informações

ID Authenticus: P-00X-5FG

Abstract (EN): This paper presents the main contributions of the VCMI Team to the ImageCLEFmedical Caption 2023 task. We addressed both the concept detection and caption prediction tasks. Regarding concept detection, our team employed different approaches to assign concepts to medical images: multi-label classification, adversarial training, autoregressive modelling, image retrieval, and concept retrieval. We also developed three model ensembles merging the results of some of the proposed methods. Our best submission obtained an F1-score of 0.4998, ranking 3rd among nine teams. Regarding the caption prediction task, our team explored two main approaches based on image retrieval and language generation. The language generation approaches, based on a vision model as the encoder and a language model as the decoder, yielded the best results, allowing us to rank 5th among thirteen teams, with a BERTScore of 0.6147. © 2023 Copyright for this paper by its authors.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Nº de páginas: 14