Empirical Software Engineering

, Volume 24, Issue 1, pp 170–207 | Cite as

An empirical study of game reviews on the Steam platform

  • Dayi LinEmail author
  • Cor-Paul Bezemer
  • Ying Zou
  • Ahmed E. Hassan


The steadily increasing popularity of computer games has led to the rise of a multi-billion dollar industry. Due to the scale of the computer game industry, developing a successful game is challenging. In addition, prior studies show that gamers are extremely hard to please, making the quality of games an important issue. Most online game stores allow users to review a game that they bought. Such reviews can make or break a game, as other potential buyers often base their purchasing decisions on the reviews of a game. Hence, studying game reviews can help game developers better understand user concerns, and further improve the user-perceived quality of games. In this paper, we perform an empirical study of the reviews of 6224 games on the Steam platform, one of the most popular digital game delivery platforms, to better understand if game reviews share similar characteristics with mobile app reviews, and thereby understand whether the conclusions and tools from mobile app review studies can be leveraged by game developers. In addition, new insights from game reviews could possibly open up new research directions for research of mobile app reviews. We first conduct a preliminary study to understand the number of game reviews and the complexity to read through them. In addition, we study the relation between several game-specific characteristics and the fluctuations of the number of reviews that are received on a daily basis. We then focus on the useful information that can be acquired from reviews by studying the major concerns that users express in their reviews, and the amount of play time before players post a review. We find that game reviews are different from mobile app reviews along several aspects. Additionally, the number of playing hours before posting a review is a unique and helpful attribute for developers that is not found in mobile app reviews. Future longitudinal studies should be conducted to help developers and researchers leverage this information. Although negative reviews contain more valuable information about the negative aspects of the game, such as mentioned complaints and bug reports, developers and researchers should also not ignore the potentially useful information in positive reviews. Our study on game reviews serves as a starting point for other game review researchers, and suggests that prior studies on mobile app reviews may need to be revisited.


Game reviews Computer games Steam 



We are grateful to Sergey Galyonkin, the owner of Steam Spy, who generously gave us access to all the historical data of Steam collected by Steam Spy for this research.


  1. Alden (2017) Steamworks Partner Program., (last visited: Mar 29, 2018)
  2. Apple Inc. (2017) Choosing a Membership., (last visited: Mar 29, 2018)
  3. Bates D, Mächler M, Bolker B, Walker S (2015) Fitting linear mixed-effects models using lme4. J Stat Softw 67(1):1–48CrossRefGoogle Scholar
  4. Becker R, Chernihov Y, Shavitt Y, Zilberman N (2012) An analysis of the Steam community network evolution. In: Proceedings of the 27th Convention of Electrical & Electronics Engineers in Israel (IEEEI). IEEE, pp 1–5Google Scholar
  5. Blackburn J, Kourtellis N, Skvoretz J, Ripeanu M, Iamnitchi A (2014) Cheating in online games: a social network perspective. ACM Trans Internet Technol (TOIT) 13(3):9CrossRefGoogle Scholar
  6. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet Allocation. J Mach Learn Res 3(Jan):993–1022zbMATHGoogle Scholar
  7. Chambers C, Feng Wc, Sahu S, Saha D (2005) Measurement-based characterization of a collection of on-line games. In: Proceedings of the 5th ACM SIGCOMM conference on Internet Measurement, USENIX Association, pp 1–1Google Scholar
  8. Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering (ICSE). ACM, New York, pp 767–778Google Scholar
  9. Cheung GK, Zimmermann T, Nagappan N (2014) The first hour experience: How the initial play can engage (or lose) new players. In: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play (CHI PLAY). ACM, New York, pp 57–66Google Scholar
  10. Ciurumelea A, Schaufelbühl A, Panichella S, Gall HC (2017) Analyzing reviews and code of mobile apps for better release planning. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, pp 91–102Google Scholar
  11. Cobbett R (2017) From Shareware superstars to the Steam gold rush: How indie conquered the PC. (last visited: Mar 29 2018)
  12. Coleman M, Liau TL (1975) A computer readability formula designed for machine scoring. J Appl Psychol 60(2):283CrossRefGoogle Scholar
  13. Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 499–510Google Scholar
  14. Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: Making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 1276–1284Google Scholar
  15. Genc-Nayebi N, Abran A (2017) A systematic literature review: Opinion mining studies from mobile app store user reviews. J Syst Softw (JSS) 125:207–219CrossRefGoogle Scholar
  16. Google (2017) How to use the Play Console., (last visited: Mar 29, 2018)
  17. Grano G, Di Sorbo A, Mercaldo F, Visaggio CA, Canfora G, Panichella S (2017) Android apps and user feedback: a dataset for software evolution and quality improvement. In: Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics. ACM, pp 8–11Google Scholar
  18. Gu X, Kim S (2015) What parts of your apps are loved by users?. In: 30Th international conference on automated software engineering (ASE). IEEE, pp 760–770Google Scholar
  19. Hassan S, Tantithamthavorn C, Bezemer CP, Hassan AE (2018) Studying the dialogue between users and developers of free apps in the google play store. Empir Softw Eng 23(3):1275–1312CrossRefGoogle Scholar
  20. Hoon L, Vasa R, Schneider JG, Mouzakis K (2012) A preliminary analysis of vocabulary in mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 245–248Google Scholar
  21. Iacob C, Harrison R (2013) Retrieving and analyzing mobile apps feature requests from online reviews. In: 10Th working conference on mining software repositories (MSR). IEEE, pp 41–44Google Scholar
  22. Khalid H, Shihab E, Nagappan M, Hassan AE (2015) What do mobile app users complain about? IEEE Softw 32(3):70–77CrossRefGoogle Scholar
  23. Kincaid J (1975) Derivation of New Readability Formulas: (automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel Research Branch report 8–75, Chief of Naval Technical Training, Naval Air Station MemphisGoogle Scholar
  24. Lewis C, Whitehead J, Wardrip-fruin N (2010) What went wrong: a taxonomy of video game bugs. In: Proceedings of the 5th International Conference on the Foundations of Digital Games (FDG). ACM, pp 108–115Google Scholar
  25. Lin D, Bezemer CP, Hassan AE (2017a) Studying the urgent updates of popular games on the Steam platform. Empir Softw Eng 22(4):2095–2126CrossRefGoogle Scholar
  26. Lin D, Bezemer CP, Hassan AE (2017b) An empirical study of early access games on the Steam platform. Empirical Software Engineering 23(2):771–799CrossRefGoogle Scholar
  27. Long JD, Feng d, Cliff N (2003) Ordinal Analysis of Behavioral Data. Wiley, HobokenCrossRefGoogle Scholar
  28. Maalej W, Nabil H (2015) Bug report, feature request, or simply praise? on automatically classifying app reviews.In: 23Rd international requirements engineering conference (RE). IEEE, pp 116–125Google Scholar
  29. Machkovech S (2018) Nintendo: letting our fans review video games might not be a good idea., (last visited: Mar 29, 2018)
  30. Man Y, Gao C, Lyu MR, Jiang J (2016) Experience Report: Understanding cross-platform app issues from user reviews. In: 27Th international symposium on software reliability engineering (ISSRE). IEEE, pp 138–149Google Scholar
  31. Martens D, Johann T (2017) On the emotion of users in app reviews. In: Proceedings of the 2nd International Workshop on Emotion Awareness in Software Engineering. IEEE Press, pp 8–14Google Scholar
  32. Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng (TSE) PP(99):1–32Google Scholar
  33. McIlroy S, Ali N, Khalid H, Hassan AE (2016) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng 21(3):1067–1106CrossRefGoogle Scholar
  34. McIlroy S, Shang W, Ali N, Hassan A (2017) Is it worth responding to reviews? a case study of the top free apps in the Google Play store. IEEE Softw 34 (3):64–71CrossRefGoogle Scholar
  35. McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189CrossRefGoogle Scholar
  36. Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, New York, pp 466–476Google Scholar
  37. Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of android apps. Empir Softw Eng 22(6):3088–3116CrossRefGoogle Scholar
  38. Orland K (2014) Introducing Steam Gauge: Ars Reveals Steam’s most popular games. (last visited: Mar 29 2018)
  39. Pagano D, Maalej W (2013) User feedback in the appstore: An empirical study. In: 21St international requirements engineering conference (RE). IEEE, pp 125–134Google Scholar
  40. Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th International Conference on Software Engineering (ICSE). IEEE Press, pp 106–117Google Scholar
  41. Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 281–290Google Scholar
  42. Pavel Djundik MB (2016) Steam DB - Steam Database., (last visited: Mar 29, 2018)
  43. SuperData Research (2016) Market Brief — Year in Review 2016., (last visited: Mar 29, 2018)
  44. Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In; Annual meeting of the Southern Association for Institutional ResearchGoogle Scholar
  45. Ruiz IJM, Nagappan M, Adams B, Berger T, Dienst S, Hassan AE (2016) Examining the rating system used in mobile-app stores. IEEE Softw 33(6):86–92CrossRefGoogle Scholar
  46. Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng (TSE) 25(4):557–572CrossRefGoogle Scholar
  47. Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008) Defect categorization: making use of a decade of widely varying historical data. In: Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM, pp 149–157Google Scholar
  48. Sergey G (2016) SteamSpy - All the data and stats about Steam games., (last visited: Mar 29, 2018)
  49. Sifa R, Drachen A, Bauckhage C (2015) Large-scale cross-game player behavior analysis on steam. In: Artificial Intelligence and Interactive Digital Entertainment International Conference. AAAI Press, pp 198–204Google Scholar
  50. Stern C (2012) what makes a game indie: a universal definition. n/, (last visited: Mar 29, 2018)Google Scholar
  51. Takahashi D (2016) PwC: Game Industry to grow nearly 5% annually through 2020. (last visited: Mar 29 2018)
  52. Valve (2016a) Steam Community., (last visited: Mar 29, 2018)
  53. Valve (2016b) Steam Store., (last visited: Mar 29, 2018)
  54. Vasa R, Hoon L, Mouzakis K, Noguchi A (2012) A preliminary analysis of mobile app user reviews. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. ACM, pp 241–244Google Scholar
  55. Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, pp 14–24Google Scholar
  56. Washburn Jr, M, Sathiyanarayanan P, Nagappan M, Zimmermann T, Bird C (2016) What went right and what went wrong: An analysis of 155 postmortems from game development. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). IEEE/ACM, pp 280–289Google Scholar
  57. Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bullet 1 (6):80–83CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Software Analysis and Intelligence Lab (SAIL)Queen’s UniversityKingstonCanada
  2. 2.Department of Electrical and Computer EngineeringQueen’s UniversityKingstonCanada

Personalised recommendations