Predicting argument density from multiple annotations

• Type: Article in International Conference Proceedings Book
• Date: 2022

Authors

Rocha, Gil (Author) Other; Leite, Bernardo (Author) Other; Trigo, Luís (Author) FLUP; Cardoso, Henrique Lopes (Author) FEUP; Sousa-Silva, Rui (Author) FLUP; Carvalho, Paula (Author) Other; Martins, Bruno (Author) Other; Won, Miguel (Author) Other

Conference proceedings International

Title: Natural Language Processing and Information Systems (NLDB 2022) : Proceedings / 27th International Conference on Applications of Natural Language to Information Systems

Pages: 227-239

27th International Conference on Applications of Natural Language to Information Systems (NLDB 2022)

Valencia, 2022

Indexing

ISI Web of Knowledge - 1 Citation

Scopus

Other information

Authenticus ID: P-00W-RH0

DOI: 10.1007/978-3-031-08473-7_21

Abstract (EN): Annotating a corpus with argument structures is a complex task, and it is even more challenging when addressing text genres where argumentative discourse markers do not abound. We explore a corpus of opinion articles annotated by multiple annotators, providing diverse perspectives of the argumentative content therein. New annotation aggregation methods are explored, diverging from the traditional ones that try to minimize presumed errors from annotator disagreement. The impact of our methods is assessed for the task of argument density prediction, seen as an initial step in the argument mining pipeline. We evaluate and compare models trained for this regression task in different generated datasets, considering their prediction error and also from a ranking perspective. Results confirm the expectation that addressing argument density from a ranking perspective is more promising than looking at the problem as a mere regression task. We also show that probabilistic aggregation, which weighs tokens by considering all annotators, is a more interesting approach, achieving encouraging results as it accommodates different annotator perspectives. The code and models are publicly available at https://github.com/DARGMINTS/argument-density.

Language: English

Type (Professor's evaluation): Scientific