Tokenizing micro-blogging messages using a text classification approach

• Type: Article in International Conference Proceedings Book
• Date: 2010

Authors

laboreiro, g (Author) Other; sarmento, l (Author) Other; teixeira, j (Author) Other; oliveira, e (Author) FEUP

Conference proceedings International

Title: International Conference on Information and Knowledge Management, Proceedings

Pages: 81-87

Proceedings of the Fourth Workshop on Analytics for Noisy Unstructured Text Data, AND 2010, Toronto, Ontario, Canada, October 26th, 2010 (in conjunction with CIKM 2010)

Indexing

ISI Web of Knowledge

Scopus - 1 Citation

Other information

Authenticus ID: P-007-WYN

DOI: 10.1145/1871840.1871853

Abstract (EN): The automatic processing of microblogging messages may be problematic, even in the case of very elementary operations such as tokenization. The problems arise from the use of non-standard language, including media-specific words (e.g. "2day", "gr8", "tl;dr", "loool"), emoticons (e.g. "(ò-ó)", "(=∧-∧=)"), non-standard letter casing (e.g. "dr. Fred") and unusual punctuation (e.g. "⋯. ..", "!!?", "","). Additionally, spelling errors are abundant (e.g. "I;m"), and we can frequently find more than one language (with different tokenization requirements) in the same short message. For being efficient in such environment, manually-developed rule-based tokenizer systems have to deal with many conditions and exceptions, which makes them difficult to build and maintain. We present a text classification approach for tokenizing Twitter messages, which address complex cases successfully and which is relatively simple to set up and maintain. For that, we created a corpus consisting of 2500 manually tokenized Twitter messages - a task that is simple for human annotators - and we trained an SVM classifier for separating tokens at certain discontinuity characters. For comparison, we created a baseline rule-based system designed specifically for dealing with typical problematic situations. Results show that we can achieve F-measures of 96% with the classification-based approach, much above the performance obtained by the baseline rule-based tokenizer (85%). Also, subsequent analysis allowed us to identify typical tokenization errors, which we show that can be partially solved by adding some additional descriptive examples to the training corpus and re-training the classifier. © 2010 ACM.

Language: English

Type (Professor's evaluation): Scientific

No. of pages: 7