Detecting Concepts and Generating Captions from Medical Images: Contributions of the VCMI Team to ImageCLEFmedical Caption 2023

• Type: Article in International Conference Proceedings Book
• Date: 2023

Authors

Torto, IR (Author) Other; Patrício, C (Author) Other; Montenegro, H (Author) Other; Gonçalves, T (Author) Other; Jaime S Cardoso (Author) FEUP

Conference proceedings International

Title: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023.

Pages: 1653-1667

24th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF-WN 2023

Thessaloniki, 18 September 2023 through 21 September 2023

Indexing

Scopus - 1 Citation

Other information

Authenticus ID: P-00X-5FG

Abstract (EN): This paper presents the main contributions of the VCMI Team to the ImageCLEFmedical Caption 2023 task. We addressed both the concept detection and caption prediction tasks. Regarding concept detection, our team employed different approaches to assign concepts to medical images: multi-label classification, adversarial training, autoregressive modelling, image retrieval, and concept retrieval. We also developed three model ensembles merging the results of some of the proposed methods. Our best submission obtained an F1-score of 0.4998, ranking 3rd among nine teams. Regarding the caption prediction task, our team explored two main approaches based on image retrieval and language generation. The language generation approaches, based on a vision model as the encoder and a language model as the decoder, yielded the best results, allowing us to rank 5th among thirteen teams, with a BERTScore of 0.6147. © 2023 Copyright for this paper by its authors.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 14