Skip to main content
SpringerLink
Log in

Categorizing the Content of GitHub README Files

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

README files play an essential role in shaping a developer’s first impression of a software repository and in documenting the software project that the repository hosts. Yet, we lack a systematic understanding of the content of a typical README file as well as tools that can process these files automatically. To close this gap, we conduct a qualitative study involving the manual annotation of 4,226 README file sections from 393 randomly sampled GitHub repositories and we design and evaluate a classifier and a set of features that can categorize these sections automatically. We find that information discussing the ‘What’ and ‘How’ of a repository is very common, while many README files lack information regarding the purpose and status of a repository. Our multi-label classifier which can predict eight different categories achieves an F1 score of 0.746. To evaluate the usefulness of the classification, we used the automatically determined classes to label sections in GitHub README files using badges and showed files with and without these badges to twenty software professionals. The majority of participants perceived the automated labeling of sections based on our classifier to ease information discovery. This work enables the owners of software repositories to improve the quality of their documentation and it has the potential to make it easier for the software development community to discover relevant information in GitHub README files.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://octoverse.github.com/

  2. https://betta.io/blog/2017/02/07/developer-experience-github-readmes/

  3. https://help.github.com/articles/about-readmes/

  4. https://guides.github.com/activities/hello-world/

  5. https://github.com/open-source

  6. https://github.com/collections/open-source-organizations

  7. https://help.github.com/articles/about-readmes/

  8. https://guides.github.com/features/mastering-markdown/

  9. https://github.com/d3/d3

  10. https://octoverse.github.com/

  11. We only consider README.md files in our work since these are the ones that GitHub initializes automatically. GitHub also supports further formats such as README.rst, but these are much less common and out of scope for this study.

  12. https://guides.github.com/features/mastering-markdown/

  13. In cases where there was perfect agreement between the two annotators, the majority vote rule simply yields the codes that both annotators agreed on.

  14. https://github.com/microlv/prerender

  15. https://github.com/jmilleralpine/ParallelGit

  16. https://github.com/solomance/sandstorm

  17. https://groups.google.com/

  18. https://github.com/ChadLactaoen/Blackjack

  19. The linguistic patterns are available in https://github.com/gprana/READMEClassifier/blob/master/doc/Patterns.ods.

  20. Original: https://github.com/readmes/alt-blog.github.io/blob/master/README1.md, Modified: https://github.com/readmes/alt-blog.github.io/blob/master/README2.md

References

  • Abebe SL, Ali N, Hassan AE (2016) An empirical study of software release notes. Empir Softw Eng 21(3):1107–1142

    Article  Google Scholar 

  • Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: Proceedings of the International Conference on Management of Data. ACM, New York, pp 207–216

  • Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc YG (2008) Is it a bug or an enhancement?: A text-based approach to classify change requests. In: Proceedings of the Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds. ACM, New York, pp 23:304–23:318

  • Asaduzzaman M, Mashiyat AS, Roy CK, Schneider KA (2013) Answering questions about unanswered questions of stack overflow. In: Proceedings of the 10th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 97–100

  • Begel A, Bosch J, Storey MA (2013) Social networking meets software development: Perspectives from GitHub, MSDN, stack exchange, and topcoder. IEEE Softw 30(1):52–66

    Article  Google Scholar 

  • Bird S, Klein E, Loper E (2009) Natural language processing with Python: Analyzing text with the natural language toolkit. O’Reilly Media Inc, Sebastopol

    MATH  Google Scholar 

  • Boughorbel S, Jarray F, El-Anbari M (2017) Optimal classifier for imbalanced data using matthews correlation coefficient metric. PloS one 12(6):e0177,678

    Article  Google Scholar 

  • Campos EC, de Almeida Maia M (2014) Automatic categorization of questions from Q&A sites. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing. ACM, New York, pp 641–643

  • Canfora G, De Lucia A, Di Penta M, Oliveto R, Panichella A, Panichella S (2013) Multi-objective cross-project defect prediction. In: 2013 IEEE 6th International Conference on Software Testing, Verification And Validation (ICST). IEEE, pp 252-261

  • Chaparro O, Lu J, Zampetti F, Moreno L, Di Penta M, Marcus A, Bavota G, Ng V (2017) Detecting missing information in bug descriptions. In: Proceedings of the Joint Meeting on Foundations of Software Engineering, ACM, pp 396–407

  • Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  MATH  Google Scholar 

  • Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) Ar-miner: Mining Informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering. ACM, New York, pp 767–778

  • Corbin JM, Strauss A (1990) Grounded theory research: Procedures, canons, and evaluative criteria. Qual Sociol 13(1):3–21

    Article  Google Scholar 

  • Correa D, Sureka A (2014) Chaff from the wheat: Characterization and modeling of deleted questions on stack overflow. In: Proceedings of the 23rd International Conference on World Wide Web. ACM, New York, pp 631–642

  • Davies S, Roper M (2014) What’s in a bug report?. In: Proceedings of the International Symposium on Empirical Software Engineering and Measurement, ACM, p 26

  • Decan A, Mens T, Claes M, Grosjean P (2016) When GitHub meets CRAN: An analysis of inter-repository package dependency problems. In: Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering. IEEE, Piscataway, pp 493–504

  • Ding W, Liang P, Tang A, Van Vliet H (2014) Knowledge-based approaches in software documentation: A systematic literature review. Inf Softw Technol 56(6):545–567

    Article  Google Scholar 

  • Erdem A, Johnson WL, Marsella S (1998) Task oriented software understanding. In: Proceedings of the 13th International Conference on Automated Software Engineering. IEEE Computer Society, Washington, DC, pp 230–239

  • Erdös K, Sneed HM (1998) Partial comprehension of complex programs (enough to perform maintenance). In: Proceedings of the 6th International Workshop on Program Comprehension. IEEE Computer Society, Washington, DC, pp 98–105

  • Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27 (8):861–874

    Article  MathSciNet  Google Scholar 

  • Fogel K (2005) Producing open source software: How to run a successful free software project. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

  • Fritz T, Murphy GC (2010) Using information fragments to answer the questions developers ask. In: Proceedings of the International Conference on Software Engineering, vol 1. ACM, New York, pp 175–184

  • Greene GJ, Fischer B (2016) Cvexplorer: Identifying candiyear developers by mining and exploring their open source contributions. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, New York, pp 804–809

  • Guzman E, El-Haliby M, Bruegge B (2015) Ensemble methods for app review classification: An approach for software evolution (n). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 771–776

  • Haenni N, Lungu M, Schwarz N, Nierstrasz O (2013) Categorizing developer information needs in software ecosystems. In: Proceedings of the International Workshop on Ecosystem Architectures. ACM, New York, pp 1–5

  • Hassan F, Wang X (2017) Mining readme files to support automatic building of Java projects in software repositories: Poster. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, Piscataway, pp 277–279

  • Hauff C, Gousios G (2015) Matching GitHub developer profiles to job advertisements. In: Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 362–366

  • Herbsleb JD, Kuwana E (1993) Preserving knowledge in design projects: What designers need to know. In: Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems. ACM, New York, pp 7–14

  • Hou D, Wong K, Hoover HJ (2005) What can programmer questions tell us about frameworks?. In: Proceedings of the 13th International Workshop on Program Comprehension. IEEE, Piscataway, pp 87–96

  • Jeong SY, Xie Y, Beaton J, Myers BA, Stylos J, Ehret R, Karstens J, Efeoglu A, Busse DK (2009) Improving documentation for eSOA APIs through user studies. In: Proceedings of the 2nd International Symposium on End-User Development. Springer, Berlin, pp 86–105

  • Johnson WL, Erdem A (1997) Interactive explanation of software systems. Autom Softw Eng 4(1):53–75

    Article  Google Scholar 

  • Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 92–101

  • Kim S, Whitehead Jr EJ, Zhang Y (2008) Classifying software changes: Clean or buggy? IEEE Trans Softw Eng 34(2):181–196

  • Kirk D, Roper M, Wood M (2007) Identifying and addressing problems in object-oriented framework reuse. Empir Softw Eng 12(3):243–274

    Article  Google Scholar 

  • Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, pp 344–353

  • Kumar N, Devanbu PT (2016) Ontocat: Automatically categorizing knowledge in API documentation. arXiv:1607.07602:preprint

  • Kurtanović Z, Maalej W (2017) Mining user rationale from software reviews. In: Proceedings of the 25th International Requirements Engineering Conference. IEEE, Piscataway, pp 61–70

  • Landis J, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/10.2307/2529310

    Article  MATH  Google Scholar 

  • LaToza TD, Myers BA (2010) Hard-to-answer questions about code. In: Evaluation and Usability of Programming Languages and Tools. ACM, New York, pp 8:1–8:6

  • Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496

    Article  Google Scholar 

  • Luaces O, Díez J, Barranquero J, del Coz JJ, Bahamonde A (2012) Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence 1(4):303–313

    Article  Google Scholar 

  • Maalej W, Robillard MP (2013) Patterns of knowledge in API reference documentation. IEEE Trans Softw Eng 39(9):1264–1282

    Article  Google Scholar 

  • Maalej W, Kurtanović Z, Nabil H, Stanik C (2016) On the automatic classification of app reviews. Requir Eng 21(3):311–331

    Article  Google Scholar 

  • Mahmoud A, Williams G (2016) Detecting, classifying, and tracing non-functional software requirements. Requir Eng 21(3):357–381

    Article  Google Scholar 

  • Miles MB, Huberman AM (1994) Qualitative data analysis: An expanded sourcebook. SAGE publications, Thousand Oaks

    Google Scholar 

  • Monperrus M, Eichberg M, Tekes E, Mezini M (2012) What should developers be aware of? an empirical study on the directives of api documentation. Empir Softw Eng 17(6):703–737

    Article  Google Scholar 

  • Moreno L, Bavota G, Di Penta M, Oliveto R, Marcus A, Canfora G (2014) Automatic generation of release notes. In: Proceedings of the International Symposium on Foundations of Software Engineering, ACM, pp 484–495

  • Mylopoulos J, Borgida A, Yu E (1997) Representing software engineering knowledge. Autom Softw Eng 4(3):291–317

    Article  Google Scholar 

  • Nam J, Pan SJ, Kim S (2013) Transfer defect learning. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 382-391

  • Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example?: A study of programming Q&A in StackOverflow. In: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society, Washington, DC, pp 25–34

  • Nykaza J, Messinger R, Boehme F, Norman CL, Mace M, Gordon M (2002) What programmers really want: Results of a needs assessment for sdk documentation. In: Proceedings of the 20th Annual International Conference on Computer Documentation. ACM, New York, pp 133–141

  • Pagano D, Maalej W (2013) How do open source communities blog? Empir Softw Eng 18(6):1090–1124

    Article  Google Scholar 

  • Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can i improve my app? classifying user reviews for software maintenance and evolution. In: 2015 IEEE international conference on Software maintenance and evolution (ICSME). IEEE, pp 281-290

  • Parnin C, Treude C (2011) Measuring API documentation on the web. In: Proceedings of the 2nd International Workshop on Web 2.0 for Software Engineering. ACM, New York, pp 25–30

  • Parnin C, Treude C, Storey MA (2013) Blogging developer knowledge: Motivations, challenges, and future directions. In: Proceedings of the 21st International Conference on Program Comprehension. IEEE Press, Piscataway, pp 211–214

  • Pascarella L, Bacchelli A (2017) Classifying code comments in java open-source software systems. In: Proceedings of the 14th International Conference on Mining Software Repositories. IEEE Press, Piscataway, pp 227–237

  • Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: Machine learning in python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

  • Portugal RLQ, do Prado Leite JCS (2016) Extracting requirements patterns from software repositories. In: Proceedings of the 24th International Requirements Engineering Conference Workshops. IEEE, Piscataway, pp 304–307

  • Prasetyo PK, Lo D, Achananuparp P, Tian Y, Lim EP (2012) Automatic classification of software related microblogs. In: Software Maintenance (ICSM), 2012 28th IEEE International Conference on, IEEE, pp 596–599

  • Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: 2013 Proceedings of the International Conference on Software Engineering. IEEE Press, pp 432-441

  • Rahman F, Posnett D, Devanbu P (2012) Recalling the imprecision of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, ACM, pp 61:1–61:11

  • Romano D, Pinzger M (2011) Using source code metrics to predict change-prone java interfaces. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE, pp 303–312

  • Sharma A, Thung F, Kochhar PS, Sulistya A, Lo D (2017) Cataloging GitHub repositories. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. ACM, New York, pp 314–319

  • Sillito J, Murphy GC, De Volder K (2006) Questions programmers ask during software evolution tasks. In: Proceedings of the International Symposium on the Foundations of Software Engineering. ACM, New York, pp 23–34

  • Sillito J, Murphy GC, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451

    Article  Google Scholar 

  • Sorbo AD, Panichella S, Visaggio CA, Penta MD, Canfora G, Gall HC (2015) Development emails content analyzer: Intention mining in developer discussions (t). In: Proceedings of the 30th International Conference on Automated Software Engineering. IEEE Press, Piscataway, pp 12–23

  • de Souza LBL, Campos EC, Maia MdA (2014) Ranking crowd knowledge to assist software development. In: Proceedings of the 22nd International Conference on Program Comprehension. ACM, New York, pp 72–82

  • Steinmacher I, Conte TU, Treude C, Gerosa MA (2016) Overcoming open source project entry barriers with a portal for newcomers. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 273–284

  • Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18

    Article  Google Scholar 

  • Tiarks R, Maalej W (2014) How does a typical tutorial for mobile development look like?. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, New York, pp 272–281

  • Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: Proceedings of the 38th International Conference on Software Engineering. ACM, New York, pp 392–403

  • Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web? (NIER track). In: Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, pp 804–807

  • Treude C, Figueira Filho F, Kulesza U (2015) Summarizing and measuring development activity. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. ACM, New York, pp 625–636

  • Trockman A, Zhou S, Kästner C, Vasilescu B (2018) Adding sparkle to social coding: an empirical study of repository badges in the npm ecosystem. In: Proceedings of the 40th International Conference on Software Engineering, ACM, pp 511–522

  • Xia X, Feng Y, Lo D, Chen Z, Wang X (2014) Towards more accurate multi-label software behavior learning. In: 2014 Software Evolution Week-IEEE Conference on Software maintenance, reengineering and reverse engineering (CSMR-WCRE). IEEE, pp 134-143

  • Zhang Y, Lo D, Kochhar PS, Xia X, Li Q, Sun J (2017) Detecting similar repositories on GitHub. In: Proceedings of the 24th International Conference on Software Analysis, Evolution and Reengineering. IEEE, Piscataway, pp 13–23

  • Zimmermann T, Premraj R, Bettenburg N, Just S, Schröter A, Weiss C (2010) What makes a good bug report? IEEE Trans Softw Eng 36(5):618–643

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Singapore Management University, Singapore, Singapore

    Gede Artha Azriadi Prana, Ferdian Thung & David Lo

  2. University of Adelaide, Adelaide, Australia

    Christoph Treude & Thushari Atapattu

Authors
  1. Gede Artha Azriadi Prana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Treude
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ferdian Thung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thushari Atapattu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Lo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gede Artha Azriadi Prana.

Additional information

Communicated by: Romain Robbes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prana, G.A.A., Treude, C., Thung, F. et al. Categorizing the Content of GitHub README Files. Empir Software Eng 24, 1296–1327 (2019). https://doi.org/10.1007/s10664-018-9660-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9660-3

Keywords

Search

Navigation