Evaluation of Lyrics Extraction from Folk Music Sheets Using Vision Language Models (VLMs)

• Type: Article in International Conference Proceedings Book
• Date: 2025

Authors

Mendes, AS (Author) Other; Murciego, AL (Author) Other; Silva, LA (Author) Other; Jiménez-Bravo, DM (Author) Other; Navarro-Cáceres, M (Author) Other; Bernardes, G (Author) FEUP

Conference proceedings International

Title: PROGRESS IN ARTIFICIAL INTELLIGENCE, EPIA 2024, PT I

Pages: 91-102

23rd EPIA Conference on Artificial Intelligence-EPIA

Viana do Castelo, PORTUGAL, SEP 03-06, 2024

Indexing

ISI Web of Knowledge - 0 Citations

Scopus - 0 Citations

Other information

Authenticus ID: P-017-C9T

DOI: 10.1007/978-3-031-73497-7_8

Abstract (EN): Monodic folk music has traditionally been preserved in physical documents. It constitutes a vast archive that needs to be digitized to facilitate comprehensive analysis using AI techniques. A critical component of music score digitization is the transcription of lyrics, an extensively researched process in Optical Character Recognition (OCR) and document layout analysis. These fields typically require the development of specific models that operate in several stages: first, to detect the bounding boxes of specific texts, then to identify the language, and finally, to recognize the characters. Recent advances in vision language models (VLMs) have introduced multimodal capabilities, such as processing images and text, which are competitive with traditional OCR methods. This paper proposes an end-to-end system for extracting lyrics from images of handwritten musical scores. We aim to evaluate the performance of two state-of-the-art VLMs to determine whether they can eliminate the need to develop specialized text recognition and OCR models for this task. The results of the study, obtained from a dataset in a real-world application environment, are presented along with promising new research directions in the field. This progress contributes to preserving cultural heritage and opens up new possibilities for global analysis and research in folk music.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 12