Advertisement

Categorizing the Content of GitHub README Files

  • Gede Artha Azriadi Prana
  • Christoph Treude
  • Ferdian Thung
  • Thushari Atapattu
  • David Lo
Article

Abstract

README files play an essential role in shaping a developer’s first impression of a software repository and in documenting the software project that the repository hosts. Yet, we lack a systematic understanding of the content of a typical README file as well as tools that can process these files automatically. To close this gap, we conduct a qualitative study involving the manual annotation of 4,226 README file sections from 393 randomly sampled GitHub repositories and we design and evaluate a classifier and a set of features that can categorize these sections automatically. We find that information discussing the ‘What’ and ‘How’ of a repository is very common, while many README files lack information regarding the purpose and status of a repository. Our multi-label classifier which can predict eight different categories achieves an F1 score of 0.746. To evaluate the usefulness of the classification, we used the automatically determined classes to label sections in GitHub README files using badges and showed files with and without these badges to twenty software professionals. The majority of participants perceived the automated labeling of sections based on our classifier to ease information discovery. This work enables the owners of software repositories to improve the quality of their documentation and it has the potential to make it easier for the software development community to discover relevant information in GitHub README files.

Keywords

GitHub README files Classification Documentation 

Notes

References

  1. Abebe SL, Ali N, Hassan AE (2016) An empirical study of software release notes. Empir Softw Eng 21(3):1107–1142CrossRefGoogle Scholar
  2. Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: Proceedings of the International Conference on Management of Data. ACM, New York, pp 207–216Google Scholar
  3. Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc YG (2008) Is it a bug or an enhancement?: A text-based approach to classify change requests. In: Proceedings of the Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds. ACM, New York, pp 23:304–23:318Google Scholar
  4. Asaduzzaman M, Mashiyat AS, Roy CK, Schneider KA (2013) Answering questions about unanswered questions of stack overflow. In: Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 97–100Google Scholar
  5. Begel A, Bosch J, Storey MA (2013) Social networking meets software development: Perspectives from GitHub, MSDN, stack exchange, and topcoder. IEEE Softw 30(1):52–66CrossRefGoogle Scholar
  6. Bird S, Klein E, Loper E (2009) Natural language processing with Python: Analyzing text with the natural language toolkit. O’Reilly Media Inc, SebastopolzbMATHGoogle Scholar
  7. Boughorbel S, Jarray F, El-Anbari M (2017) Optimal classifier for imbalanced data using matthews correlation coefficient metric. PloS one 12(6):e0177,678CrossRefGoogle Scholar
  8. Campos EC, de Almeida Maia M (2014) Automatic categorization of questions from Q&A sites. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing. ACM, New York, pp 641–643Google Scholar
  9. Canfora G, De Lucia A, Di Penta M, Oliveto R, Panichella A, Panichella S (2013) Multi-objective cross-project defect prediction. In: 2013 IEEE 6th International Conference on Software Testing, Verification And Validation (ICST). IEEE, pp 252-261Google Scholar
  10. Chaparro O, Lu J, Zampetti F, Moreno L, Di Penta M, Marcus A, Bavota G, Ng V (2017) Detecting missing information in bug descriptions. In: Proceedings of the Joint Meeting on Foundations of Software Engineering, ACM, pp 396–407Google Scholar
  11. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357CrossRefGoogle Scholar
  12. Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering. ACM, New York, pp 767–778Google Scholar
  13. Corbin JM, Strauss A (1990) Grounded theory research: Procedures, canons, and evaluative criteria. Qual Sociol 13(1):3–21CrossRefGoogle Scholar
  14. Correa D, Sureka A (2014) Chaff from the wheat: Characterization and modeling of deleted questions on stack overflow. In: Proceedings of the 23rd International Conference on World Wide Web. ACM, New York, pp 631–642Google Scholar
  15. Davies S, Roper M (2014) What’s in a bug report?. In: Proceedings of the International Symposium on Empirical Software Engineering and Measurement, ACM, p 26Google Scholar
  16. Decan A, Mens T, Claes M, Grosjean P (2016) When GitHub meets CRAN: An analysis of inter-repository package dependency problems. In: Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering. IEEE, Piscataway, pp 493–504Google Scholar
  17. Ding W, Liang P, Tang A, Van Vliet H (2014) Knowledge-based approaches in software documentation: A systematic literature review. Inf Softw Technol 56(6):545–567CrossRefGoogle Scholar
  18. Erdem A, Johnson WL, Marsella S (1998) Task oriented software understanding. In: Proceedings of the 13th International Conference on Automated Software Engineering. IEEE Computer Society, Washington, DC, pp 230–239Google Scholar
  19. Erdös K, Sneed HM (1998) Partial comprehension of complex programs (enough to perform maintenance). In: Proceedings of the 6th International Workshop on Program Comprehension. IEEE Computer Society, Washington, DC, pp 98–105Google Scholar
  20. Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27 (8):861–874MathSciNetCrossRefGoogle Scholar
  21. Fogel K (2005) Producing open source software: How to run a successful free software project. O’Reilly Media, Inc., SebastopolGoogle Scholar
  22. Fritz T, Murphy GC (2010) Using information fragments to answer the questions developers ask. In: Proceedings of the International Conference on Software Engineering, vol 1. ACM, New York, pp 175–184Google Scholar
  23. Greene GJ, Fischer B (2016) Cvexplorer: Identifying candiyear developers by mining and exploring their open source contributions. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, New York, pp 804–809Google Scholar
  24. Guzman E, El-Haliby M, Bruegge B (2015) Ensemble methods for app review classification: An approach for software evolution (n). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 771–776Google Scholar
  25. Haenni N, Lungu M, Schwarz N, Nierstrasz O (2013) Categorizing developer information needs in software ecosystems. In: Proceedings of the International Workshop on Ecosystem Architectures. ACM, New York, pp 1–5Google Scholar
  26. Hassan F, Wang X (2017) Mining readme files to support automatic building of Java projects in software repositories: Poster. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, Piscataway, pp 277–279Google Scholar
  27. Hauff C, Gousios G (2015) Matching GitHub developer profiles to job advertisements. In: Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 362–366Google Scholar
  28. Herbsleb JD, Kuwana E (1993) Preserving knowledge in design projects: What designers need to know. In: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. ACM, New York, pp 7–14Google Scholar
  29. Hou D, Wong K, Hoover HJ (2005) What can programmer questions tell us about frameworks?. In: Proceedings of the 13th International Workshop on Program Comprehension. IEEE, Piscataway, pp 87–96Google Scholar
  30. Jeong SY, Xie Y, Beaton J, Myers BA, Stylos J, Ehret R, Karstens J, Efeoglu A, Busse DK (2009) Improving documentation for eSOA APIs through user studies. In: Proceedings of the 2nd International Symposium on End-User Development. Springer, Berlin, pp 86–105CrossRefGoogle Scholar
  31. Johnson WL, Erdem A (1997) Interactive explanation of software systems. Autom Softw Eng 4(1):53–75CrossRefGoogle Scholar
  32. Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 92–101Google Scholar
  33. Kim S, Whitehead Jr EJ, Zhang Y (2008) Classifying software changes: Clean or buggy? IEEE Trans Softw Eng 34(2):181–196Google Scholar
  34. Kirk D, Roper M, Wood M (2007) Identifying and addressing problems in object-oriented framework reuse. Empir Softw Eng 12(3):243–274CrossRefGoogle Scholar
  35. Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 344–353Google Scholar
  36. Kumar N, Devanbu PT (2016) Ontocat: Automatically categorizing knowledge in API documentation. arXiv:1607.07602:preprint
  37. Kurtanović Z, Maalej W (2017) Mining user rationale from software reviews. In: Proceedings of the 25th International Requirements Engineering Conference. IEEE, Piscataway, pp 61–70Google Scholar
  38. Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174.  https://doi.org/10.2307/2529310 CrossRefGoogle Scholar
  39. LaToza TD, Myers BA (2010) Hard-to-answer questions about code. In: Evaluation and Usability of Programming Languages and Tools. ACM, New York, pp 8:1–8:6Google Scholar
  40. Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496CrossRefGoogle Scholar
  41. Luaces O, Díez J, Barranquero J, del Coz JJ, Bahamonde A (2012) Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence 1(4):303–313CrossRefGoogle Scholar
  42. Maalej W, Robillard MP (2013) Patterns of knowledge in API reference documentation. IEEE Trans Softw Eng 39(9):1264–1282CrossRefGoogle Scholar
  43. Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Requir Eng 21(3):311–331CrossRefGoogle Scholar
  44. Mahmoud A, Williams G (2016) Detecting, classifying, and tracing non-functional software requirements. Requir Eng 21(3):357–381CrossRefGoogle Scholar
  45. Miles MB, Huberman AM (1994) Qualitative data analysis: An expanded sourcebook. SAGE publications, Thousand OaksGoogle Scholar
  46. Monperrus M, Eichberg M, Tekes E, Mezini M (2012) What should developers be aware of? an empirical study on the directives of api documentation. Empir Softw Eng 17(6):703–737CrossRefGoogle Scholar
  47. Moreno L, Bavota G, Di Penta M, Oliveto R, Marcus A, Canfora G (2014) Automatic generation of release notes. In: Proceedings of the International Symposium on Foundations of Software Engineering, ACM, pp 484–495Google Scholar
  48. Mylopoulos J, Borgida A, Yu E (1997) Representing software engineering knowledge. Autom Softw Eng 4(3):291–317CrossRefGoogle Scholar
  49. Nam J, Pan SJ, Kim S (2013) Transfer defect learning. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 382-391Google Scholar
  50. Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example?: A study of programming Q&A in StackOverflow. In: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society, Washington, DC, pp 25–34Google Scholar
  51. Nykaza J, Messinger R, Boehme F, Norman CL, Mace M, Gordon M (2002) What programmers really want: Results of a needs assessment for sdk documentation. In: Proceedings of the 20th Annual International Conference on Computer Documentation. ACM, New York, pp 133–141Google Scholar
  52. Pagano D, Maalej W (2013) How do open source communities blog? Empir Softw Eng 18(6):1090–1124CrossRefGoogle Scholar
  53. Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: 2015 IEEE international conference on Software maintenance and evolution (ICSME). IEEE, pp 281-290Google Scholar
  54. Parnin C, Treude C (2011) Measuring API documentation on the web. In: Proceedings of the 2nd International Workshop on Web 2.0 for Software Engineering. ACM, New York, pp 25–30Google Scholar
  55. Parnin C, Treude C, Storey MA (2013) Blogging developer knowledge: Motivations, challenges, and future directions. In: Proceedings of the 21st International Conference on Program Comprehension. IEEE Press, Piscataway, pp 211–214Google Scholar
  56. Pascarella L, Bacchelli A (2017) Classifying code comments in java open-source software systems. In: Proceedings of the 14th International Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 227–237Google Scholar
  57. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine learning in python. J Mach Learn Res 12(Oct):2825–2830MathSciNetzbMATHGoogle Scholar
  58. Portugal RLQ, do Prado Leite JCS (2016) Extracting requirements patterns from software repositories. In: Proceedings of the 24th International Requirements Engineering Conference Workshops. IEEE, Piscataway, pp 304–307Google Scholar
  59. Prasetyo PK, Lo D, Achananuparp P, Tian Y, Lim EP (2012) Automatic classification of software related microblogs. In: Software Maintenance (ICSM), 2012 28th IEEE International Conference on, IEEE, pp 596–599Google Scholar
  60. Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 432-441Google Scholar
  61. Rahman F, Posnett D, Devanbu P (2012) Recalling the imprecision of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ACM, pp 61:1–61:11Google Scholar
  62. Romano D, Pinzger M (2011) Using source code metrics to predict change-prone java interfaces. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, pp 303–312Google Scholar
  63. Sharma A, Thung F, Kochhar PS, Sulistya A, Lo D (2017) Cataloging GitHub repositories. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. ACM, New York, pp 314–319Google Scholar
  64. Sillito J, Murphy GC, De Volder K (2006) Questions programmers ask during software evolution tasks. In: Proceedings of the International Symposium on the Foundations of Software Engineering. ACM, New York, pp 23–34Google Scholar
  65. Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451CrossRefGoogle Scholar
  66. Sorbo AD, Panichella S, Visaggio CA, Penta MD, Canfora G, Gall HC (2015) Development emails content analyzer: Intention mining in developer discussions (t). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 12–23Google Scholar
  67. de Souza LBL, Campos EC, Maia MdA (2014) Ranking crowd knowledge to assist software development. In: Proceedings of the 22nd International Conference on Program Comprehension. ACM, New York, pp 72–82Google Scholar
  68. Steinmacher I, Conte TU, Treude C, Gerosa MA (2016) Overcoming open source project entry barriers with a portal for newcomers. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 273–284Google Scholar
  69. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18CrossRefGoogle Scholar
  70. Tiarks R, Maalej W (2014) How does a typical tutorial for mobile development look like?. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 272–281Google Scholar
  71. Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 392–403Google Scholar
  72. Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web? (NIER track). In: Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, pp 804–807Google Scholar
  73. Treude C, Figueira Filho F, Kulesza U (2015) Summarizing and measuring development activity. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. ACM, New York, pp 625–636Google Scholar
  74. Trockman A, Zhou S, Kästner C, Vasilescu B (2018) Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In: Proceedings of the 40th International Conference on Software Engineering, ACM, pp 511–522Google Scholar
  75. Xia X, Feng Y, Lo D, Chen Z, Wang X (2014) Towards more accurate multi-label software behavior learning. In: 2014 Software Evolution Week-IEEE Conference on Software maintenance, reengineering and reverse engineering (CSMR-WCRE). IEEE, pp 134-143Google Scholar
  76. Zhang Y, Lo D, Kochhar PS, Xia X, Li Q, Sun J (2017) Detecting similar repositories on GitHub. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering. IEEE, Piscataway, pp 13–23Google Scholar
  77. Zimmermann T, Premraj R, Bettenburg N, Just S, Schröter A, Weiss C (2010) What makes a good bug report? IEEE Trans Softw Eng 36(5):618–643CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Singapore Management UniversitySingaporeSingapore
  2. 2.University of AdelaideAdelaideAustralia

Personalised recommendations