Categorizing the Content of GitHub README Files

Abstract

README files play an essential role in shaping a developer’s first impression of a software repository and in documenting the software project that the repository hosts. Yet, we lack a systematic understanding of the content of a typical README file as well as tools that can process these files automatically. To close this gap, we conduct a qualitative study involving the manual annotation of 4,226 README file sections from 393 randomly sampled GitHub repositories and we design and evaluate a classifier and a set of features that can categorize these sections automatically. We find that information discussing the ‘What’ and ‘How’ of a repository is very common, while many README files lack information regarding the purpose and status of a repository. Our multi-label classifier which can predict eight different categories achieves an F1 score of 0.746. To evaluate the usefulness of the classification, we used the automatically determined classes to label sections in GitHub README files using badges and showed files with and without these badges to twenty software professionals. The majority of participants perceived the automated labeling of sections based on our classifier to ease information discovery. This work enables the owners of software repositories to improve the quality of their documentation and it has the potential to make it easier for the software development community to discover relevant information in GitHub README files.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    https://octoverse.github.com/

  2. 2.

    https://betta.io/blog/2017/02/07/developer-experience-github-readmes/

  3. 3.

    https://help.github.com/articles/about-readmes/

  4. 4.

    https://guides.github.com/activities/hello-world/

  5. 5.

    https://github.com/open-source

  6. 6.

    https://github.com/collections/open-source-organizations

  7. 7.

    https://help.github.com/articles/about-readmes/

  8. 8.

    https://guides.github.com/features/mastering-markdown/

  9. 9.

    https://github.com/d3/d3

  10. 10.

    https://octoverse.github.com/

  11. 11.

    We only consider README.md files in our work since these are the ones that GitHub initializes automatically. GitHub also supports further formats such as README.rst, but these are much less common and out of scope for this study.

  12. 12.

    https://guides.github.com/features/mastering-markdown/

  13. 13.

    In cases where there was perfect agreement between the two annotators, the majority vote rule simply yields the codes that both annotators agreed on.

  14. 14.

    https://github.com/microlv/prerender

  15. 15.

    https://github.com/jmilleralpine/ParallelGit

  16. 16.

    https://github.com/solomance/sandstorm

  17. 17.

    https://groups.google.com/

  18. 18.

    https://github.com/ChadLactaoen/Blackjack

  19. 19.

    The linguistic patterns are available in https://github.com/gprana/READMEClassifier/blob/master/doc/Patterns.ods.

  20. 20.

    Original: https://github.com/readmes/alt-blog.github.io/blob/master/README1.md, Modified: https://github.com/readmes/alt-blog.github.io/blob/master/README2.md

References

  1. Abebe SL, Ali N, Hassan AE (2016) An empirical study of software release notes. Empir Softw Eng 21(3):1107–1142

    Article  Google Scholar 

  2. Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: Proceedings of the International Conference on Management of Data. ACM, New York, pp 207–216

  3. Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc YG (2008) Is it a bug or an enhancement?: A text-based approach to classify change requests. In: Proceedings of the Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds. ACM, New York, pp 23:304–23:318

  4. Asaduzzaman M, Mashiyat AS, Roy CK, Schneider KA (2013) Answering questions about unanswered questions of stack overflow. In: Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 97–100

  5. Begel A, Bosch J, Storey MA (2013) Social networking meets software development: Perspectives from GitHub, MSDN, stack exchange, and topcoder. IEEE Softw 30(1):52–66

    Article  Google Scholar 

  6. Bird S, Klein E, Loper E (2009) Natural language processing with Python: Analyzing text with the natural language toolkit. O’Reilly Media Inc, Sebastopol

    Google Scholar 

  7. Boughorbel S, Jarray F, El-Anbari M (2017) Optimal classifier for imbalanced data using matthews correlation coefficient metric. PloS one 12(6):e0177,678

    Article  Google Scholar 

  8. Campos EC, de Almeida Maia M (2014) Automatic categorization of questions from Q&A sites. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing. ACM, New York, pp 641–643

  9. Canfora G, De Lucia A, Di Penta M, Oliveto R, Panichella A, Panichella S (2013) Multi-objective cross-project defect prediction. In: 2013 IEEE 6th International Conference on Software Testing, Verification And Validation (ICST). IEEE, pp 252-261

  10. Chaparro O, Lu J, Zampetti F, Moreno L, Di Penta M, Marcus A, Bavota G, Ng V (2017) Detecting missing information in bug descriptions. In: Proceedings of the Joint Meeting on Foundations of Software Engineering, ACM, pp 396–407

  11. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    MATH  Article  Google Scholar 

  12. Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering. ACM, New York, pp 767–778

  13. Corbin JM, Strauss A (1990) Grounded theory research: Procedures, canons, and evaluative criteria. Qual Sociol 13(1):3–21

    Article  Google Scholar 

  14. Correa D, Sureka A (2014) Chaff from the wheat: Characterization and modeling of deleted questions on stack overflow. In: Proceedings of the 23rd International Conference on World Wide Web. ACM, New York, pp 631–642

  15. Davies S, Roper M (2014) What’s in a bug report?. In: Proceedings of the International Symposium on Empirical Software Engineering and Measurement, ACM, p 26

  16. Decan A, Mens T, Claes M, Grosjean P (2016) When GitHub meets CRAN: An analysis of inter-repository package dependency problems. In: Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering. IEEE, Piscataway, pp 493–504

  17. Ding W, Liang P, Tang A, Van Vliet H (2014) Knowledge-based approaches in software documentation: A systematic literature review. Inf Softw Technol 56(6):545–567

    Article  Google Scholar 

  18. Erdem A, Johnson WL, Marsella S (1998) Task oriented software understanding. In: Proceedings of the 13th International Conference on Automated Software Engineering. IEEE Computer Society, Washington, DC, pp 230–239

  19. Erdös K, Sneed HM (1998) Partial comprehension of complex programs (enough to perform maintenance). In: Proceedings of the 6th International Workshop on Program Comprehension. IEEE Computer Society, Washington, DC, pp 98–105

  20. Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27 (8):861–874

    MathSciNet  Article  Google Scholar 

  21. Fogel K (2005) Producing open source software: How to run a successful free software project. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

  22. Fritz T, Murphy GC (2010) Using information fragments to answer the questions developers ask. In: Proceedings of the International Conference on Software Engineering, vol 1. ACM, New York, pp 175–184

  23. Greene GJ, Fischer B (2016) Cvexplorer: Identifying candiyear developers by mining and exploring their open source contributions. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, New York, pp 804–809

  24. Guzman E, El-Haliby M, Bruegge B (2015) Ensemble methods for app review classification: An approach for software evolution (n). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 771–776

  25. Haenni N, Lungu M, Schwarz N, Nierstrasz O (2013) Categorizing developer information needs in software ecosystems. In: Proceedings of the International Workshop on Ecosystem Architectures. ACM, New York, pp 1–5

  26. Hassan F, Wang X (2017) Mining readme files to support automatic building of Java projects in software repositories: Poster. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, Piscataway, pp 277–279

  27. Hauff C, Gousios G (2015) Matching GitHub developer profiles to job advertisements. In: Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 362–366

  28. Herbsleb JD, Kuwana E (1993) Preserving knowledge in design projects: What designers need to know. In: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. ACM, New York, pp 7–14

  29. Hou D, Wong K, Hoover HJ (2005) What can programmer questions tell us about frameworks?. In: Proceedings of the 13th International Workshop on Program Comprehension. IEEE, Piscataway, pp 87–96

  30. Jeong SY, Xie Y, Beaton J, Myers BA, Stylos J, Ehret R, Karstens J, Efeoglu A, Busse DK (2009) Improving documentation for eSOA APIs through user studies. In: Proceedings of the 2nd International Symposium on End-User Development. Springer, Berlin, pp 86–105

  31. Johnson WL, Erdem A (1997) Interactive explanation of software systems. Autom Softw Eng 4(1):53–75

    Article  Google Scholar 

  32. Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 92–101

  33. Kim S, Whitehead Jr EJ, Zhang Y (2008) Classifying software changes: Clean or buggy? IEEE Trans Softw Eng 34(2):181–196

  34. Kirk D, Roper M, Wood M (2007) Identifying and addressing problems in object-oriented framework reuse. Empir Softw Eng 12(3):243–274

    Article  Google Scholar 

  35. Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 344–353

  36. Kumar N, Devanbu PT (2016) Ontocat: Automatically categorizing knowledge in API documentation. arXiv:1607.07602:preprint

  37. Kurtanović Z, Maalej W (2017) Mining user rationale from software reviews. In: Proceedings of the 25th International Requirements Engineering Conference. IEEE, Piscataway, pp 61–70

  38. Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/10.2307/2529310

    MATH  Article  Google Scholar 

  39. LaToza TD, Myers BA (2010) Hard-to-answer questions about code. In: Evaluation and Usability of Programming Languages and Tools. ACM, New York, pp 8:1–8:6

  40. Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496

    Article  Google Scholar 

  41. Luaces O, Díez J, Barranquero J, del Coz JJ, Bahamonde A (2012) Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence 1(4):303–313

    Article  Google Scholar 

  42. Maalej W, Robillard MP (2013) Patterns of knowledge in API reference documentation. IEEE Trans Softw Eng 39(9):1264–1282

    Article  Google Scholar 

  43. Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Requir Eng 21(3):311–331

    Article  Google Scholar 

  44. Mahmoud A, Williams G (2016) Detecting, classifying, and tracing non-functional software requirements. Requir Eng 21(3):357–381

    Article  Google Scholar 

  45. Miles MB, Huberman AM (1994) Qualitative data analysis: An expanded sourcebook. SAGE publications, Thousand Oaks

    Google Scholar 

  46. Monperrus M, Eichberg M, Tekes E, Mezini M (2012) What should developers be aware of? an empirical study on the directives of api documentation. Empir Softw Eng 17(6):703–737

    Article  Google Scholar 

  47. Moreno L, Bavota G, Di Penta M, Oliveto R, Marcus A, Canfora G (2014) Automatic generation of release notes. In: Proceedings of the International Symposium on Foundations of Software Engineering, ACM, pp 484–495

  48. Mylopoulos J, Borgida A, Yu E (1997) Representing software engineering knowledge. Autom Softw Eng 4(3):291–317

    Article  Google Scholar 

  49. Nam J, Pan SJ, Kim S (2013) Transfer defect learning. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 382-391

  50. Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example?: A study of programming Q&A in StackOverflow. In: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society, Washington, DC, pp 25–34

  51. Nykaza J, Messinger R, Boehme F, Norman CL, Mace M, Gordon M (2002) What programmers really want: Results of a needs assessment for sdk documentation. In: Proceedings of the 20th Annual International Conference on Computer Documentation. ACM, New York, pp 133–141

  52. Pagano D, Maalej W (2013) How do open source communities blog? Empir Softw Eng 18(6):1090–1124

    Article  Google Scholar 

  53. Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: 2015 IEEE international conference on Software maintenance and evolution (ICSME). IEEE, pp 281-290

  54. Parnin C, Treude C (2011) Measuring API documentation on the web. In: Proceedings of the 2nd International Workshop on Web 2.0 for Software Engineering. ACM, New York, pp 25–30

  55. Parnin C, Treude C, Storey MA (2013) Blogging developer knowledge: Motivations, challenges, and future directions. In: Proceedings of the 21st International Conference on Program Comprehension. IEEE Press, Piscataway, pp 211–214

  56. Pascarella L, Bacchelli A (2017) Classifying code comments in java open-source software systems. In: Proceedings of the 14th International Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 227–237

  57. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine learning in python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

  58. Portugal RLQ, do Prado Leite JCS (2016) Extracting requirements patterns from software repositories. In: Proceedings of the 24th International Requirements Engineering Conference Workshops. IEEE, Piscataway, pp 304–307

  59. Prasetyo PK, Lo D, Achananuparp P, Tian Y, Lim EP (2012) Automatic classification of software related microblogs. In: Software Maintenance (ICSM), 2012 28th IEEE International Conference on, IEEE, pp 596–599

  60. Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 432-441

  61. Rahman F, Posnett D, Devanbu P (2012) Recalling the imprecision of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ACM, pp 61:1–61:11

  62. Romano D, Pinzger M (2011) Using source code metrics to predict change-prone java interfaces. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, pp 303–312

  63. Sharma A, Thung F, Kochhar PS, Sulistya A, Lo D (2017) Cataloging GitHub repositories. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. ACM, New York, pp 314–319

  64. Sillito J, Murphy GC, De Volder K (2006) Questions programmers ask during software evolution tasks. In: Proceedings of the International Symposium on the Foundations of Software Engineering. ACM, New York, pp 23–34

  65. Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451

    Article  Google Scholar 

  66. Sorbo AD, Panichella S, Visaggio CA, Penta MD, Canfora G, Gall HC (2015) Development emails content analyzer: Intention mining in developer discussions (t). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 12–23

  67. de Souza LBL, Campos EC, Maia MdA (2014) Ranking crowd knowledge to assist software development. In: Proceedings of the 22nd International Conference on Program Comprehension. ACM, New York, pp 72–82

  68. Steinmacher I, Conte TU, Treude C, Gerosa MA (2016) Overcoming open source project entry barriers with a portal for newcomers. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 273–284

  69. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18

    Article  Google Scholar 

  70. Tiarks R, Maalej W (2014) How does a typical tutorial for mobile development look like?. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 272–281

  71. Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 392–403

  72. Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web? (NIER track). In: Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, pp 804–807

  73. Treude C, Figueira Filho F, Kulesza U (2015) Summarizing and measuring development activity. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. ACM, New York, pp 625–636

  74. Trockman A, Zhou S, Kästner C, Vasilescu B (2018) Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In: Proceedings of the 40th International Conference on Software Engineering, ACM, pp 511–522

  75. Xia X, Feng Y, Lo D, Chen Z, Wang X (2014) Towards more accurate multi-label software behavior learning. In: 2014 Software Evolution Week-IEEE Conference on Software maintenance, reengineering and reverse engineering (CSMR-WCRE). IEEE, pp 134-143

  76. Zhang Y, Lo D, Kochhar PS, Xia X, Li Q, Sun J (2017) Detecting similar repositories on GitHub. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering. IEEE, Piscataway, pp 13–23

  77. Zimmermann T, Premraj R, Bettenburg N, Just S, Schröter A, Weiss C (2010) What makes a good bug report? IEEE Trans Softw Eng 36(5):618–643

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Gede Artha Azriadi Prana.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Romain Robbes

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Prana, G.A.A., Treude, C., Thung, F. et al. Categorizing the Content of GitHub README Files. Empir Software Eng 24, 1296–1327 (2019). https://doi.org/10.1007/s10664-018-9660-3

Download citation

Keywords

  • GitHub README files
  • Classification
  • Documentation