Can Llama 3 Accurately Assess Readability? A Comparative Study Using Lead Sections from Wikipedia

• Type: Article in International Conference Proceedings Book
• Date: 2025

Authors

Rodrigues, JF (Author) Other; Cardoso, HL (Author) Other; Carla Teixeira Lopes (Author) FEUP

Conference proceedings International

Title: RESEARCH CHALLENGES IN INFORMATION SCIENCE, RCIS 2025, PT II

Pages: 89-97

19th International Conference on Research Challenges in Information Science-RCIS-Annual

Sevilla, SPAIN, MAY 20-23, 2025

Indexing

ISI Web of Knowledge - 0 Citations

Scopus - 0 Citations

Other information

Authenticus ID: P-018-RZ8

DOI: 10.1007/978-3-031-92471-2_6

Abstract (EN): Text readability is vital for effective communication and learning, especially for those with lower information literacy. This research aims to assess Llama 3's ability to grade readability and compare its alignment with established metrics. For that purpose, we create a new dataset of article lead sections from English and Simple English Wikipedia, covering nine categories. The model is prompted to rate the readability of the texts on a grade-level scale, and an in-depth analysis of the results is conducted. While Llama 3 correlates strongly with most metrics, it may underestimate text grade levels.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 9