Skip to main content
SpringerLink
Log in

An Enriched Emoji Picker to Improve Accessibility in Mobile Communications

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2021 (INTERACT 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12932))

Included in the following conference series:

  • 2378 Accesses

Abstract

We present an emoji picker designed to enrich emojis selection on mobile devices using audio cues. The aim is to make emojis selection more intuitive by better identify their meanings. Unlike the typical emoji input components currently in use (known as “pickers”), in our component each emotion-related item is represented by both an emoji and a non-verbal vocal cue, and it is displayed according to a two-dimensional model suggesting the pleasantness and intensity of the emotion itself. The component was embedded in an Android app in order to exploit touchscreen interaction together with audio cues to ease the selection process by using more than one channel (visual and auditory). Since the component adds non-visual information that drives the emoji selection, it may be particularly useful for users with visual impairments. In order to investigate the feasibility of the approach and the acceptability/usability of the emoji picker component, a preliminary remote evaluation test involving both sighted and visually impaired users was performed. Analysis of the data collected through the evaluation test shows that all the participants, whether sighted or visually impaired, rated the usability of our picker as good, and also evaluated positively the model adopted to add semantic value to emojis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)

    Article  Google Scholar 

  2. Yik, M., Russell, J.A., Steiger, J.H.: A 12-point circumplex structure of core affect. Emotion 11(4), 705–731 (2011)

    Article  Google Scholar 

  3. Walther, J., Tidwell, L.C.: Nonverbal cues in computer-mediated communication, and the effect of chronemics on relational communication. J. Organ. Comput. 5, 355–378 (1995)

    Google Scholar 

  4. Tigwell, G.W., Gorman, B.M., Menzies, R.: Emoji accessibility for visually impaired people. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020)

    Google Scholar 

  5. Toet, A., Erp, J.V.: The EmojiGrid as a tool to assess experienced and perceived emotions. Psychology 1, 469–481 (2019)

    Google Scholar 

  6. Derks, D., Bos, A.E., Grumbkow, J.V.: Emoticons in computer-mediated communication: social motives and social context. Cycberpsychol. Behav. 11(1), 99–101 (2008)

    Article  Google Scholar 

  7. Völkel, S.T., Buschek, D., Pranjic, J., Hußmann, H.: Understanding emoji interpretation through user personality and message context. In: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (2019)

    Google Scholar 

  8. Walther, J., D’Addario, K.P.: The impacts of emoticons on message interpretation in computer-mediated communication. Soc. Sci. Comput. Rev. 19, 324–347 (2001)

    Article  Google Scholar 

  9. Liebman, N., Gergle, D.: It’s (not) simply a matter of time: the relationship between cmc cues and interpersonal affinity. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing (2016)

    Google Scholar 

  10. Cramer, H., Juan, P.D., Tetreault, J.: Sender-intended functions of emojis in US messaging. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (2016)

    Google Scholar 

  11. Unicode Inc.: Unicode® Full Emoji List, v13.1. https://www.unicode.org/reports/tr51/ (2020). Accessed: 12 October 2020

  12. http://unicode.org/emoji/charts/full-emoji-list.html

  13. Barbieri, F., Kruszewski, G., Ronzano, F., Saggion, H.: How cosmopolitan are emojis? Exploring emojis usage and meaning over different languages with distributional semantics. In: Proceedings of the 24th ACM international conference on Multimedia (2016)

    Google Scholar 

  14. Lu, X., Ai, W., Liu, X., Li, Q., Wang, N., Huang, G., Mei, Q.: Learning from the ubiquitous language: an empirical analysis of emoji usage of smartphone users. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. (2016)

    Google Scholar 

  15. Belin, P., Fillion-Bilodeau, S., Gosselin, F.: The Montreal affective voices: a validated set of nonverbal affect bursts for research on auditory affective processing. Behav. Res. Methods 40, 531–539 (2008)

    Article  Google Scholar 

  16. Cowen, A., Elfenbein, H., Laukka, P., Keltner, D.: Mapping 24 emotions conveyed by brief human vocalization. Am. Psychol. 74(6), 698–712 (2019). https://doi.org/10.1037/amp0000399

    Article  Google Scholar 

  17. https://s3-us-west-1.amazonaws.com/vocs/map.html

  18. Posner, J.D., Russell, J., Peterson, B.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005)

    Article  Google Scholar 

  19. Banse, R., Scherer, K.: Acoustic profiles in vocal emotion expression. J. Pers. Soc. Psychol. 70(3), 614–636 (1996)

    Article  Google Scholar 

  20. Schröder, M.: Experimental study of affect bursts. Speech Commun. 40, 99–116 (2003)

    Article  Google Scholar 

  21. Updated Emoji Statistics: https://www.emojipedia.org/stats/

  22. Zhong, K., Qiao, T., Zhang, L.: A study of emotional communication of emoticon based on Russell’s circumplex model of affect. In: HCI (2019)

    Google Scholar 

  23. Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I.L., Terveen, L., Hecht, B.J.: “Blissfully happy” or “ready to fight”: varying interpretations of emoji. In: ICWSM (2016)

    Google Scholar 

  24. Alismail, S., Zhang, H.: Exploring and understanding participants’ perceptions of facial emoji Likert scales in online surveys. ACM Trans. Soc. Comput. 3, 1–12 (2020)

    Article  Google Scholar 

  25. Herring, S., Dainas, A.: Gender and age influences on interpretation of emoji functions. ACM Trans. Soc. Comput. 3, 1–26 (2020)

    Article  Google Scholar 

  26. Brooke, J.: SUS: a “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (eds.) Usability Evaluation in Industry. Taylor and Francis, London (1986)

    Google Scholar 

  27. Brooke, J.B.: SUS—a retrospective. J. Usabil. Stud. 8(2), 29–40 (2013)

    Google Scholar 

  28. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971)

    Article  Google Scholar 

  29. Bangor, A., Kortum, P.T., Miller, J.T.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usabil. Stud. Arch. 4, 114–123 (2009)

    Google Scholar 

  30. Sirait, E.R., Zellatifanny, C.M.: An empirical study: computer-mediated communication and collaboration among government employees during flexible working arrangements. In: 2020 International Conference on Information Technology Systems and Innovation (ICITSI), pp. 95–100 (2020)

    Google Scholar 

  31. Runge, N., Hellmeier, M., Wenig, D., Malaka, R.: Tag your emotions: a novel mobile user interface for annotating images with emotions. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (2016)

    Google Scholar 

  32. Warpechowski, K., Orzeszek, D., Nielek, R.: Tagging emotions using a wheel user interface. In: Proceedings of the 13th Biannual Conference of the Italian SIGCHI Chapter: Designing the Next Interaction (2019)

    Google Scholar 

  33. Sacharin, V., Schlegel, K., Scherer, K.: Geneva Emotion Wheel Rating Study. https://archive-ouverte.unige.ch/unige:97849 (2012)

  34. Frequency of Emoji Use: https://home.unicode.org/emoji/emoji-frequency/

Download references

Acknowledgments

We wish to thank the following email groups devoted to accessibility and users with visual impairments, who showed particular interest in our study and helped us find participants: tech-vi@groups.io, accessible@googlemail.com. We wish to also thank the Blind & Visually Impaired Android Users on Facebook.

Author information

Authors and Affiliations

  1. ISTI – CNR, via Moruzzi 1, 56124, Pisa, Italy

    Maria Teresa Paratore & Barbara Leporini

  2. IIT – CNR, via Moruzzi 1, 56124, Pisa, Italy

    Maria Claudia Buzzi & Marina Buzzi

Authors
  1. Maria Teresa Paratore
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Claudia Buzzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marina Buzzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Barbara Leporini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Claudia Buzzi .

Editor information

Editors and Affiliations

  1. Department of Electrical and Information Engineering, Polytechnic University of Bari, Bari, Italy

    Carmelo Ardito

  2. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Rosa Lanzilotti

  3. Computer Science Department, University of Pisa, Pisa, Italy

    Alessio Malizia

  4. Department of Computer Science, University of York, York, UK

    Helen Petrie

  5. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  6. Computer Science Department, University of Bari Aldo Moro, Bari, Italy

    Giuseppe Desolda

  7. Microsoft Research, Redmond, WA, USA

    Kori Inkpen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Paratore, M.T., Buzzi, M.C., Buzzi, M., Leporini, B. (2021). An Enriched Emoji Picker to Improve Accessibility in Mobile Communications. In: Ardito, C., et al. Human-Computer Interaction – INTERACT 2021. INTERACT 2021. Lecture Notes in Computer Science(), vol 12932. Springer, Cham. https://doi.org/10.1007/978-3-030-85623-6_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85623-6_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85622-9

  • Online ISBN: 978-3-030-85623-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation