Human Experts vs. Large Language Models: Evaluating Annotation Scheme and Guidelines Development for Clinical

• Tipo: Artigo em Livro de Atas de Conferência Internacional
• Data: 2025

Autores

Fernandes, Ana Luísa Cardoso (Autor) Outra; Silvano, Maria da Purificação (Autor) FLUP; Guimaraes, Nuno (Autor) FEUP; Rb-Silva, Rita (Autor) Outra

Ata de Conferência Internacional

Título: Proceedings of Text2Story - Eighth Workshop on Narrative Extraction From Texts held in conjunction with the 47th European Conference on Information Retrieval (ECIR 2025)

Text2Story — Eighth Workshop on Narrative Extraction From Texts held in conjunction with the 47th European Conference on Information Retrieval (ECIR 2025)

Lucca, 2025

Classificação Científica

CORDIS: Humanidades > Ciências da linguagem > Linguística

Outras Informações

Abstract (EN): Electronic Health Records (EHRs) contain vast amounts of unstructured narrative text, posing challenges for organization, curation, and automated information extraction in clinical and research settings. Developing e"ective annotation schemes is crucial for training extraction models, yet it remains complex for both human experts and Large Language Models (LLMs). This study compares human- and LLM-generated annotation schemes and guidelines through an experimental framework. In the !rst phase, both a human expert and an LLM created annotation schemes based on prede!ned criteria. In the second phase, experienced annotators applied these schemes following the guidelines. In both cases, the results were qualitatively evaluated using Likert scales. The !ndings indicate that the human-generated scheme is more comprehensive, coherent, and clear compared to those produced by the LLM. These results align with previous research suggesting that while LLMs show promising performance with respect to text annotation, the same does not apply to the development of annotation schemes, and human validation remains essential to ensure accuracy and reliability.

Idioma: Inglês

Tipo (Avaliação Docente): Científica

Notas: Também participaram neste artigo os seguintes autores: Tahsir Ahmed Munna1, Filipe Cunha1, António Leal, Ricardo Campos e Alípio Jorge.

Nº de páginas: 11