A Case Study on using Crowdsourcing for Ambiguous Tasks

Ankush Chatterjee, Umang Gupta, Puneet Agrawal

2018

Abstract

In our day to day life, we come across situations which are interpreted differently by different human beings. A given sentence may be offensive to some humans but not to others. Similarly, a sentence can convey different emotions to different human beings. For instance, “Why you never text me!”, can either be interpreted as a sad or an angry utterance. Lack of facial expressions and voice modulations make detecting emotions in textual sentences a hard problem. Some textual sentences are inherently ambiguous and their true emotion label is difficult to determine. In this paper, we study how to use crowdsourcing for an ambiguous task of determining emotion labels of textual sentences. Crowdsourcing has become one of the most popular medium for obtaining large scale labeled data for supervised learning tasks. However, for our task, due to the intrinsic ambiguity, human annotators differ in opinions about the underlying emotion of certain sentences. In our work, we harness the multiple perspectives of annotators for ambiguous sentences to improve the performance of an emotion detection model. In particular, we compare our technique against the popularly used technique of majority vote to determine the label of a given sentence. Our results indicate that considering diverse perspective of annotators is helpful for the ambiguous task of emotion detection.

Paper Citation

in Harvard Style

Chatterjee A., Gupta U. and Agrawal P. (2018). A Case Study on using Crowdsourcing for Ambiguous Tasks. In Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI; ISBN 978-989-758-327-8, SciTePress, pages 242-247. DOI: 10.5220/0006955002420247

in Bibtex Style

@conference{ijcci18,
author={Ankush Chatterjee and Umang Gupta and Puneet Agrawal},
title={A Case Study on using Crowdsourcing for Ambiguous Tasks},
booktitle={Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI},
year={2018},
pages={242-247},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006955002420247},
isbn={978-989-758-327-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Joint Conference on Computational Intelligence (IJCCI 2018) - Volume 1: IJCCI
TI - A Case Study on using Crowdsourcing for Ambiguous Tasks
SN - 978-989-758-327-8
AU - Chatterjee A.
AU - Gupta U.
AU - Agrawal P.
PY - 2018
SP - 242
EP - 247
DO - 10.5220/0006955002420247
PB - SciTePress