Skip to main content
SpringerLink
Log in

Suggestions for Online User Studies

Sharing Experiences from the Use of Four Platforms

  • Conference paper
  • First Online:
HCI International 2021 - Late Breaking Papers: Design and User Experience (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13094))

Included in the following conference series:

Abstract

During exceptional times when researchers do not have physical access to users of technology, the importance of remote user studies increases. We provide recommendations based on lessons learned from conducting online user studies utilizing four online research platforms (Appen, MTurk, Prolific, and Upwork). Our recommendations aim to help those inexperienced with online user studies. They are also beneficial for those interested in increasing their proficiency, employing this increasingly important research methodology for studying people’s interactions with technology and information.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.appen.com (retrieved January 11, 2021).

  2. 2.

    https://forums.aws.amazon.com/thread.jspa?threadID=58891.

  3. 3.

    https://prolific.ac.

  4. 4.

    https://www.prnewswire.com/news-releases/snagajob-appoints-former-upwork-ceo-to-board-of-directors-300417689.html.

  5. 5.

    https://www.surveysystem.com/sscalc.htm.

  6. 6.

    https://researcher-help.prolific.co/hc/en-gb/articles/360019236753-Representative-Samples-on-Prolific.

  7. 7.

    https://www.gov.uk/government/publications/the-national-minimum-wage-in-2020.

References

  1. Ford, D., et al.: A Tale of Two Cities: Software Developers Working from Home During the COVID-19 Pandemic. arXiv:2008.11147 [cs]. (2020)

  2. Yee, N., Bailenson, J.N., Rickertsen, K.: A meta-analysis of the impact of the inclusion and realism of human-like faces on user experiences in interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1–10 (2007)

    Google Scholar 

  3. Toms, E.G., Freund, L., Li, C.: WiIRE: the Web interactive information retrieval experimentation system prototype. Inf. Process. Manag. 40, 655–675 (2004). https://doi.org/10.1016/j.ipm.2003.08.006

    Article  MATH  Google Scholar 

  4. Simons, R.N., Gurari, D., Fleischmann, K.R.: I hope this is helpful: understanding crowdworkers’ challenges and motivations for an image description task. Proc. ACM Hum.-Comput. Interact. 4, 105:1–105:26 (2020). https://doi.org/10.1145/3415176

  5. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 203–212 (2010)

    Google Scholar 

  6. Komarov, S., Reinecke, K., Gajos, K.Z.: Crowdsourcing performance evaluations of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 207–216. Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2470654.2470684

  7. Barbosa, N.M., Chen, M.: Rehumanized crowdsourcing: a labeling framework addressing bias and ethics in machine learning. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019)

    Google Scholar 

  8. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Proceedings of Eleventh International AAAI Conference on Web and Social Media, pp. 512–515. Montreal, Canada (2017)

    Google Scholar 

  9. Weber, I., Mejova, Y.: Crowdsourcing health labels: inferring body weight from profile pictures. In: Proceedings of the 6th International Conference on Digital Health Conference, pp. 105–109 (2016)

    Google Scholar 

  10. Quinn, A.J., Bederson, B.B.: Human computation: a survey and taxonomy of a growing field. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1403–1412 (2011)

    Google Scholar 

  11. Gadiraju, U., Kawase, R., Dietze, S., Demartini, G.: Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1631–1640 (2015)

    Google Scholar 

  12. Ikeda, K., Bernstein, M.S.: Pay it backward: Per-task payments on crowdsourcing platforms reduce productivity. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 4111–4121 (2016)

    Google Scholar 

  13. Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers? Shifting demographics in Mechanical Turk. In: CHI 2010 Extended Abstracts on Human Factors in Computing Systems. pp. 2863–2872 (2010)

    Google Scholar 

  14. Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with Mechanical Turk. In: Proceedings of the twenty-sixth annual SIGCHI Conference on Human Factors in Computing Systems, pp. 453–456. ACM, New York, NY, USA (2008). https://doi.org/10.1145/1357054.1357127

  15. Hill, J.R., Harrington, A.B., Adeoye, P., Campbell, N.L., Holden, R.J.: Going remote—demonstration and evaluation of remote technology delivery and usability assessment with older adults: survey study. JMIR mHealth and uHealth. 9, e26702 (2021)

    Google Scholar 

  16. Cook, M.: Evidence-based medicine and experience-based practice-clash or consensus. Med. L. 23, 735 (2004)

    Google Scholar 

  17. Kozleski, E.B.: The uses of qualitative research: powerful methods to inform evidence-based practice in education. Res. Pract. Pers. Sev. Disabil. 42, 19–32 (2017). https://doi.org/10.1177/1540796916683710

    Article  Google Scholar 

  18. Albarqouni, L., Hoffmann, T., Glasziou, P.: Evidence-based practice educational intervention studies: a systematic review of what is taught and how it is measured. BMC Med. Educ. 18, 177 (2018). https://doi.org/10.1186/s12909-018-1284-1

    Article  Google Scholar 

  19. Salminen, J., Jung, S., Santos, J.M., Jansen, B.J.: Does a smile matter if the person is not real? The effect of a smile and stock photos on persona perceptions Int. J. Hum.-Comput. Interact. 1–23 (2019). https://doi.org/10.1080/10447318.2019.1664068

  20. Salminen, J., Jung, S., Santos, J.M., Jansen, B.J.: The effect of smiling pictures on perceptions of personas. In: UMAP 2019 Adjunct: Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization. ACM, Larnaca, Cyprus (2019). https://doi.org/10.1145/3314183.3324973

  21. Salminen, J., Santos, J.M., Kwak, H., An, J., Jung, S., Jansen, B.J.: Persona perception scale: development and exploratory validation of an instrument for evaluating individuals’ perceptions of personas. Int. J. Hum. Comput. Stud. 141, 102437 (2020). https://doi.org/10.1016/j.ijhcs.2020.102437

    Article  Google Scholar 

  22. Salminen, J., Kwak, H., Santos, J.M., Jung, S., An, J., Jansen, B.J.: Persona perception scale: developing and validating an instrument for human-like representations of data. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems - CHI 2018, pp. 1–6. ACM Press, Montreal QC, Canada (2018). https://doi.org/10.1145/3170427.3188461

  23. Salminen, J., Jung, S., Santos, J.M., Kamel, A.M., Jansen, B.J.: Picturing it!: the effect of image styles on user perceptions of personas. In: In the Proceedings of ACM Human Factors in Computing Systems (CHI 2021). ACM, Virtual Conference (2021)

    Google Scholar 

  24. Salminen, J., Jung, S., Kamel, A.M.S., Santos, J.M., Jansen, B.J.: Using artificially generated pictures in customer-facing systems: an evaluation study with data-driven personas Behav. Inf. Technol. 1–17 (2020). https://doi.org/10.1080/0144929X.2020.1838610

  25. Salminen, J., Santos, J.M., Jung, S., Eslami, M., Jansen, B.J.: Persona transparency: analyzing the impact of explanations on perceptions of data-driven personas Int. J. Hum. Comput. Interact. 1–13 (2019). https://doi.org/10.1080/10447318.2019.1688946

  26. Jansen, B., Salminen, J., Jung, S., Guan, K.: Data-Driven Personas. Morgan & Claypool Publishers, San Rafael (2021)

    Google Scholar 

  27. Difallah, D., Filatova, E., Ipeirotis, P.: Demographics and dynamics of mechanical Turk workers. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 135–143. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3159652.3159661

  28. Sheehan, K.B., Pittman, M.: Amazon’s Mechanical Turk for academics: The HIT handbook for social science research. Melvin & Leigh, Publishers (2016)

    Google Scholar 

  29. Palan, S., Schitter, C.: Prolific. ac–a subject pool for online experiments. J. Behav. Exp. Financ. 17, 22–27 (2018)

    Google Scholar 

  30. Popiel, P.: Boundaryless in the creative economy: assessing freelancing on upwork. Crit. Stud. Med. Commun. 34, 220–233 (2017)

    Article  Google Scholar 

  31. Salminen, J., Nielsen, L., Jung, S., An, J., Kwak, H., Jansen, B.J.: Is more better? Impact of multiple photos on perception of persona profiles. In: Proceedings of ACM CHI Conference on Human Factors in Computing Systems (CHI2018). ACM, Montréal, Canada (2018). https://doi.org/10.1145/3173574.3173891

  32. Salminen, J., Jung, S., Chowdhury, S.A., Sengün, S., Jansen, B.J.: Personas and analytics: a comparative user study of efficiency and effectiveness for a user identification task. In: Proceedings of the ACM Conference of Human Factors in Computing Systems (CHI 2020). ACM, Honolulu, Hawaii, USA (2020). https://doi.org/10.1145/3313831.3376770

  33. Alonso, O., Marshall, C.C., Najork, M.A.: A human-centered framework for ensuring reliability on crowdsourced labeling tasks. In: First AAAI Conference on Human Computation and Crowdsourcing (2013)

    Google Scholar 

  34. Gleibs, I.H.: Are all research fields equal? Rethinking practice for the use of data from crowdsourcing market places. Behav. Res. Methods 49(4), 1333–1342 (2016). https://doi.org/10.3758/s13428-016-0789-y

    Article  Google Scholar 

  35. d’Eon, G., Goh, J., Larson, K., Law, E.: Paying Crowd Workers for Collaborative Work. Proc. ACM Hum.-Comput. Interact. 3, 1–24 (2019)

    Article  Google Scholar 

  36. Peer, E., Brandimarte, L., Samat, S., Acquisti, A.: Beyond the Turk: alternative platforms for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70, 153–163 (2017)

    Article  Google Scholar 

  37. Hauser, D.J., Schwarz, N.: Attentive turkers: mturk participants perform better on online attention checks than do subject pool participants. Behav. Res. Methods 48(1), 400–407 (2015). https://doi.org/10.3758/s13428-015-0578-z

    Article  Google Scholar 

  38. Jung, S., Salminen, J., Jansen, B.J.: Persona analytics: implementing mouse-tracking for an interactive persona system. In: Extended Abstracts of ACM Human Factors in Computing Systems - CHI EA 2021. ACM, Virtual conference (2021). https://doi.org/10.1145/3411763.3451773

  39. Chen, M.C., Anderson, J.R., Sohn, M.H.: What can a mouse cursor tell us more? Correlation of eye/mouse movements on web browsing. In: CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 281–282. ACM (2001)

    Google Scholar 

  40. Navalpakkam, V., Churchill, E.: Mouse tracking: measuring and predicting users’ experience of web-based content. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2963–2972 (2012)

    Google Scholar 

  41. Papoutsaki, A., Gokaslan, A., Tompkin, J., He, Y., Huang, J.: The eye of the typer: a benchmark and analysis of gaze behavior during typing. In: Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pp. 1–9. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3204493.3204552

  42. Papoutsaki, A., Laskey, J., Huang, J.: SearchGazer: Webcam eye tracking for remote studies of web search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pp. 17–26. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3020165.3020170

  43. Papoutsaki, A., Sangkloy, P., Laskey, J., Daskalova, N., Huang, J., Hays, J.: Webgazer: scalable webcam eye tracking using user interactions. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 3839–3845. AAAI Press, New York, New York, USA (2016)

    Google Scholar 

  44. Jung, S.-G., Salminen, J., Jansen, B.J.: Implementing eye-tracking for persona analytics. In: ETRA 2021 Adjunct: ACM Symposium on Eye Tracking Research and Applications, pp. 1–4. ACM, Virtual Conference (2021). https://doi.org/10.1145/3450341.3458765

  45. Lee, G., Xia, W.: A longitudinal experimental study on the interaction effects of persuasion quality, user training, and first-hand use on user perceptions of new information technology. Inf. Manag. 48, 288–295 (2011)

    Article  Google Scholar 

  46. Yuan, W.: End-user searching behavior in information retrieval: a longitudinal study. J. Am. Soc. Inf. Sci. 48, 218–234 (1997)

    Article  Google Scholar 

  47. Kothe, E., Ling, M.: Retention of participants recruited to a one-year longitudinal study via Prolific (2019). https://psyarxiv.com/5yv2u/. https://doi.org/10.31234/osf.io/5yv2u

  48. Luzhnica, G., Veas, E., Pammer, V.: Skin reading: encoding text in a 6-channel haptic display. In: Proceedings of the 2016 ACM International Symposium on Wearable Computers, pp. 148–155. ACM, New York, NY, USA (2016). https://doi.org/10.1145/2971763.2971769

  49. Shklovski, I., Vertesi, J.: Un-Googling: research technologies, communities at risk and the ethics of user studies in HCI. In: The 26th BCS Conference on Human Computer Interaction 26, pp. 1–4 (2012)

    Google Scholar 

  50. Ferguson, L.M., Yonge, O., Myrick, F.: Students’ involvement in faculty research: ethical and methodological issues. Int. J. Qual. Methods 3, 56–68 (2004)

    Article  Google Scholar 

  51. Häkkilä, J., et al.: Design sensibilities-designing for cultural sensitivity. In: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pp. 1–3 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Qatar Computing Research Institute, Hamad Bin Khalifa University, Ar-Rayyan, Qatar

    Joni Salminen, Soon-gyo Jung & Bernard J. Jansen

  2. Turku School of Economics at the University of Turku, Turku, Finland

    Joni Salminen

Authors
  1. Joni Salminen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soon-gyo Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernard J. Jansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joni Salminen .

Editor information

Editors and Affiliations

  1. University of Creteand Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Department of Design, Federal University of Pernambuco, Recife, Brazil

    Marcelo M. Soares

  3. World Usability Day and Brandeis University, Newton Center, MA, USA

    Elizabeth Rosenzweig

  4. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

  5. Tokyo University of Science, Tokyo, Saitama, Japan

    Sakae Yamamoto

  6. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  7. Tsinghua University, Beijing, China

    Pei-Luen Patrick Rau

  8. Computer and Information Sciences, Towson University, Towson, MD, USA

    Gabriele Meiselwitz

  9. DePaul University, Chicago, IL, USA

    Xiaowen Fang

  10. San Jose State University, San Jose, CA, USA

    Abbas Moallem

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Salminen, J., Jung, Sg., Jansen, B.J. (2021). Suggestions for Online User Studies. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Design and User Experience. HCII 2021. Lecture Notes in Computer Science(), vol 13094. Springer, Cham. https://doi.org/10.1007/978-3-030-90238-4_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90238-4_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90237-7

  • Online ISBN: 978-3-030-90238-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation