Go to:
Logótipo
Comuta visibilidade da coluna esquerda
Você está em: Start > Publications > View > How to evaluate sentiment classifiers for Twitter time-ordered data?
Publication

Publications

How to evaluate sentiment classifiers for Twitter time-ordered data?

Title
How to evaluate sentiment classifiers for Twitter time-ordered data?
Type
Article in International Scientific Journal
Year
2018
Authors
Mozetic, I
(Author)
Other
The person does not belong to the institution. The person does not belong to the institution. The person does not belong to the institution. Without AUTHENTICUS Without ORCID
Torgo, L
(Author)
FCUP
View Personal Page You do not have permissions to view the institutional email. Search for Participant Publications View Authenticus page Without ORCID
Cerqueira, V
(Author)
Other
View Personal Page You do not have permissions to view the institutional email. Search for Participant Publications View Authenticus page View ORCID page
Smailovic, J
(Author)
Other
The person does not belong to the institution. The person does not belong to the institution. The person does not belong to the institution. Without AUTHENTICUS Without ORCID
Journal
Title: PLoS ONEImported from Authenticus Search for Journal Publications
Vol. 13 No. 1
ISSN: 1932-6203
Other information
Authenticus ID: P-00N-R0E
Abstract (EN): Social media are becoming an increasingly important source of information about the public mood regarding issues such as elections, Brexit, stock market, etc. In this paper we focus on sentiment classification of Twitter data. Construction of sentiment classifiers is a standard text mining task, but here we address the question of how to properly evaluate them as there is no settled way to do so. Sentiment classes are ordered and unbalanced, and Twitter produces a stream of time-ordered data. The problem we address concerns the procedures used to obtain reliable estimates of performance measures, and whether the temporal ordering of the training and test data matters. We collected a large set of 1.5 million tweets in 13 European languages. We created 138 sentiment models and out-of-sample datasets, which are used as a gold standard for evaluations. The corresponding 138 in-sample data-sets are used to empirically compare six different estimation procedures: three variants of cross-validation, and three variants of sequential validation (where test set always follows the training set). We find no significant difference between the best cross-validation and sequential validation. However, we observe that all cross-validation variants tend to overestimate the performance, while the sequential methods tend to underestimate it. Standard cross-validation with random selection of examples is significantly worse than the blocked cross-validation, and should not be used to evaluate classifiers in time-ordered data scenarios.
Language: English
Type (Professor's evaluation): Scientific
No. of pages: 20
Documents
We could not find any documents associated to the publication.
Related Publications

Of the same authors

A Comparative Study of Performance Estimation Methods for Time Series Forecasting (2017)
Article in International Conference Proceedings Book
Cerqueira, V; Torgo, L; Smailovic, J; Mozetic, I

Of the same journal

Reliability in long-term clinical studies of disease-modifying therapies for relapsing-remitting multiple sclerosis: A systematic review (2020)
Another Publication in an International Scientific Journal
Lucchetta, RC; Leonart, LP; Goncalves, MVM; Becker, J; Pontarolo, R; Fernandez Llimos, F; Wiens, A
Questionnaires measuring movement behaviours in adults and older adults: Content description and measurement properties. A systematic review (2022)
Another Publication in an International Scientific Journal
Rodrigues, B; Encantado, J; Carraca, E; Sousa Sa, E; Lopes, L; Cliff, D; Mendes, R; Silva, MN; Godinho, C; Santos, R
Mapping the characteristics of network meta-analyses on drug therapy: A systematic review (2018)
Another Publication in an International Scientific Journal
Tonin, FS; Steimbach, LM; Mendes, AM; Borba, HH; Pontarolo, R; Fernandez Llimos, F
Fecal microbiota transplantation in inflammatory bowel disease patients: A systematic review and meta-analysis (2020)
Another Publication in an International Scientific Journal
Caldeira, LD; Borba, HH; Tonin, FS; Wiens, A; Fernandez Llimos, F; Pontarolo, R
Establishing the criterion validity of self-report measures of adherence in hemodialysis through associations with clinical biomarkers: A systematic review and meta-analysis (2022)
Another Publication in an International Scientific Journal
Sousa, H; Ribeiro, O; Costa, E; Christensen, AJ; Figueiredo, D

See all (436)

Recommend this page Top
Copyright 1996-2025 © Faculdade de Direito da Universidade do Porto  I Terms and Conditions  I Acessibility  I Index A-Z
Page created on: 2025-07-23 at 06:46:01 | Privacy Policy | Personal Data Protection Policy | Whistleblowing