Empirical Software Engineering

, Volume 24, Issue 1, pp 7–32 | Cite as

Studying the consistency of star ratings and reviews of popular free hybrid Android and iOS apps

  • Hanyang Hu
  • Shaowei WangEmail author
  • Cor-Paul Bezemer
  • Ahmed E. Hassan


Nowadays, many developers make their mobile apps available on multiple platforms (e.g., Android and iOS). However, maintaining several versions of a cross-platform app that is built natively (i.e., using platform-specific tools) is a complicated task. Instead, developers can choose to use hybrid development tools, such as PhoneGap, to build hybrid apps. Hybrid apps are based on a single codebase across platforms. There exist two ways to use a hybrid development tool to build a hybrid app that runs on multiple platforms: (1) using web technologies (i.e., HTML, Javascript and CSS) and (2) in a common language, which is then converted to native code. We study whether these hybrid development tools achieve their main purpose: delivering an app that is perceived similarly by users across platforms. Prior studies show that users refer to star ratings and user reviews, when deciding to download an app. Given the importance of star ratings and user reviews, we study whether the usage of a hybrid development tool assists app developers in achieving consistency in the star ratings and user reviews across multiple platforms. We study 68 hybrid app-pairs, i.e., apps that exist both in the Google Play store and Apple App store. We find that 33 out of 68 hybrid apps do not receive consistent star ratings across platforms. We run Twitter-LDA on user reviews and find that the star ratings of the reviews that discuss the same topic could be up to three times as high across platforms. Our findings suggest that while hybrid apps are better at providing consistent star ratings and user reviews when compared to cross-platform apps that are built natively, hybrid apps do not guarantee such consistency. Hence, developers should not solely rely on hybrid development tools to achieve consistency in the star ratings and reviews that are given by users of their apps. In particular, developers should track closely the ratings and reviews of their apps across platforms, so that they can act accordingly when platform-specific issues arise.


Mobile apps Star rating User reviews Twitter-LDA 


  1. Adobe (2017) Phonegap., (last visited: Oct 3, 2017)
  2. Akdeniz (2014) Google Play Crawler JAVA API., (last visited: Jan 25, 2017)
  3. Ali M, Mesbah A (2016) Mining and characterizing hybrid apps. In: Proceedings of the International Workshop on App Market Analytics (WAMA), ACM, pp 50–56Google Scholar
  4. Apple (2008) RSS feed provided by Apple for the app “Facebook”., (last visited: Jan 25, 2017)
  5. Benenson Z, Gassmann F, Reinfelder L (2013) Android and iOS users’ differences concerning security and privacy. In: Extended Abstracts on Human Factors in Computing Systems (CHI), pp 817–822Google Scholar
  6. Blei D M, Ng A Y, Jordan M I (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022zbMATHGoogle Scholar
  7. Chen N, Lin J, Hoi S C H, Xiao X, Zhang B (2014) Ar-miner: Mining informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering (ICSE). ACM, New York, pp 767–778Google Scholar
  8. Dalmasso I, Datta SK, Bonnet C, Nikaein N (2013) Survey, comparison and evaluation of cross platform mobile application development tools. In: 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC), pp 323–328Google Scholar
  9. Di Sorbo A, Panichella S, Alexandru C V, Shimagaki J, Visaggio C A, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), ACM, pp 499–510Google Scholar
  10. Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: Making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, pp 1276–1284Google Scholar
  11. Graphpad Software (2015) Interpreting results: Skewness and kurtosis., (last visited: Jan 30, 2016)
  12. Gu X, Kim S (2015) What parts of your apps are loved by users? In: 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 760–770Google Scholar
  13. Guzman E, Maalej W (2014) How do users like this feature? [a] fine grained sentiment analysis of app reviews. In: 22nd International Requirements Engineering Conference (RE), IEEE, pp 153–162Google Scholar
  14. Harman M, Jia Y, Zhang Y (2012) App store mining and analysis: MSR for app stores. In: 9th Working Conference on Mining Software Repositories (MSR), IEEE, pp 108–111Google Scholar
  15. Hassan S, Shang W, Hassan A E (2017) An empirical study of emergency updates for top Android mobile apps. Empir Softw Eng 22(1):505–546CrossRefGoogle Scholar
  16. Heitkötter H, Hanschke S, Majchrzak T A (2013) Evaluating cross-platform development approaches for mobile applications. Springer Berlin Heidelberg, Berlin, pp 120–138Google Scholar
  17. Hu H, Bezemer CP, Hassan AE (2016) Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps.
  18. Joanes D N, Gill C A (1998) Comparing measures of sample skewness and kurtosis. J R Stat Soc Ser D (Stat) 47(1):183–189CrossRefGoogle Scholar
  19. Joorabchi M, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE/ACM, pp 15–24Google Scholar
  20. Joorabchi ME, Ali M, Mesbah A (2015) Detecting inconsistencies in multi-platform mobile apps. In: IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp 450–460Google Scholar
  21. Khalid H, Shihab E, Nagappan M, Hassan A E (2015) What do mobile app users complain about IEEE Soft 32(3):70–77CrossRefGoogle Scholar
  22. Long J D, Feng D, Cliff N (2003) Ordinal analysis of behavioral data. Wiley, New YorkCrossRefGoogle Scholar
  23. Man Y, Gao C, Lyu MR, Jiang J (2016) Experience report: Understanding cross-platform app issues from user reviews. In: IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp 138–149Google Scholar
  24. Mann H B, Whitney D R (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Statist 18(1):50–60MathSciNetCrossRefzbMATHGoogle Scholar
  25. Martin W, Harman M, Jia Y, Sarro F, Zhang Y (2015) The app sampling problem for app store mining. In: Proceedings of the 12th Working Conference on Mining Software Repositories (MSR), IEEE Press, pp 123–133Google Scholar
  26. Martin W, Sarro F, Harman M (2016a) Causal impact analysis for app releases in google play. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York, pp 435–446Google Scholar
  27. Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2016b) A survey of app store analysis for software engineering. IEEE Trans. Softw. Eng. PP(99):1–32Google Scholar
  28. Microsoft (2017) Xamarin: Mobile app development and app creation software., (last visited: Oct 3, 2017)
  29. NIST/SEMATECH (2012) e-handbook of statistical methods: Measures of skewness and kurtosis., (last visited: Oct 3, 2017)
  30. Noei E, Syer M D, Zou Y, Hassan A E, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir Softw Eng 22:1–29CrossRefGoogle Scholar
  31. Ohrt J, Turau V (2012) Cross-platform development tools for smartphone applications. Computer 45(9):72–79CrossRefGoogle Scholar
  32. Pagano D, Maalej W (2013) User feedback in the appstore: An empirical study. In: 21st International Requirements Engineering Conference (RE), IEEE, pp 125–134Google Scholar
  33. Palmieri M, Singh I, Cicchetti A (2012) Comparison of cross-platform mobile development tools. In: 2012 16th International Conference on Intelligence in Next Generation Networks (ICIN), pp 179–186Google Scholar
  34. Palomba F, Linares-Vásquez M, Bavota G, Oliveto R, Penta MD, Poshyvanyk D, Lucia AD (2015) User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 291–300Google Scholar
  35. Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th International Conference on Software Engineering (ICSE). IEEE Press, Piscataway, pp 106–117Google Scholar
  36. Panichella S, Sorbo A D, Guzman E, Visaggio C A, Canfora G, Gall HC (2015) How can I improve my app? Classifying user reviews for software maintenance and evolution. In: International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 281–290Google Scholar
  37. Pettey C, Rob van der M (2012) Gartner says free apps will account for nearly 90 percent of total mobile app store downloads in 2012., (last visited: Jan 28, 2016)
  38. Porter M F (1997) Readings in information retrieval. Morgan Kaufmann Publishers Inc., chap An Algorithm for Suffix Stripping, pp 313–316Google Scholar
  39. Poschenrieder M (2015) 77% will not download a retail app rated lower than 3 stars., (last visited: Jan 28, 2016)
  40. Ramon L, Ryan R, Kathy N (2015) Smartphone OS market share, 2015 q2., (last visited: Jan 25, 2017)
  41. Romano J, Kromrey J D, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of the Southern Association for Institutional ResearchGoogle Scholar
  42. Smutný P (2012) Mobile development tools and cross-platform solutions. In: Carpathian Control Conference (ICCC), 2012 13th International, pp 653–656Google Scholar
  43. Statista (2017) Most popular Apple App Store categories in July 2017, by share of available apps.
  44. Thomas S W, Adams B, Hassan A E, Blostein D (2011) Modeling the evolution of topics in source code histories. In: Proceedings of the 8th Working Conference on Mining Software Repositories (MSR), ACM, pp 173–182Google Scholar
  45. Tian Y, Nagappan M, Lo D, Hassan AE (2015) What are the characteristics of high-rated apps? A case study on free Android applications. In: IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 301–310Google Scholar
  46. Vashistha C (2015) Native vs hybrid mobile app: 5 ways to choose right plarform.
  47. Viennot N, Garcia E, Nieh J (2014) A measurement study of google play. SIGMETRICS Perform Eval Rev 42(1):221–233CrossRefGoogle Scholar
  48. Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, New York, pp 14–24Google Scholar
  49. Vu PM, Nguyen TT, Pham HV, Nguyen TT (2015) Mining user opinions in mobile app reviews: A keyword-based approach. In: 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 749–759Google Scholar
  50. Zhao W X, Jiang J, Weng J, He J, Lim E P, Yan H, Li X (2011) Comparing Twitter and traditional media using topic models. In: Proceedings of the 33rd European Conference on Advances in Information Retrieval (ECIR), Springer-Verlag, pp 338–349CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Queen’s UniversityKingstonCanada

Personalised recommendations