World Wide Web

, Volume 22, Issue 4, pp 1699–1725 | Cite as

LinkLive: discovering Web learning resources for developers from Q&A discussions

  • Jing LiEmail author
  • Zhenchang Xing
  • Aixin Sun


Software developers need access to correlated information (e.g., API documentation, Wikipedia pages, Stack Overflow questions and answers) which are often dispersed among different Web resources. This paper is concerned with the situation where a developer is visiting a Web page, but at the same time is willing to explore correlated Web resources to extend his/her knowledge or to satisfy his/her curiosity. Specifically, we present an item-based collaborative filtering technique, named LinkLive, for automatically recommending a list of correlated Web resources for a particular Web page. The recommendation is done by exploiting hyperlink associations from the crowdsourced knowledge on Stack Overflow. We motivate our research using an exploratory study of hyperlink dissemination patterns on Stack Overflow. We then present our LinkLive technique that uses multiple features, including hyperlink co-occurrences in Q&A discussions, locations (e.g., question, answer, or comment) in which hyperlinks are referenced, and votes for posts/comments in which hyperlinks are referenced. Experiments using 7 years of Stack Overflow data show that, our technique recommends correlated Web resources with promising accuracy in an open setting. A user study of 6 participants suggests that practitioners find the recommended Web resources useful for Web discovery.


API documentation Recommendation systems Web discovery Social media 



  1. 1.
    Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. 2.
    Ahn, J.-W., Brusilovsky, P., Grady, J., He, D., Syn, S.Y.: Open user profiles for adaptive news systems: help or harm?. In: Proceedings of WWW, pp. 11–20. ACM (2007)Google Scholar
  3. 3.
    Anderson, A., Huttenlocher, D., Kleinberg, J., Leskovec, J.: Discovering value from community activity on focused question answering sites: a case study of stack overflow. In: Proceedings of KDD, pp. 850–858 (2012)Google Scholar
  4. 4.
    Ar, Y., Bostanci, E.: A genetic algorithm solution to the collaborative filtering problem. Expert Syst. Appl. 61, 122–128 (2016)CrossRefGoogle Scholar
  5. 5.
    Baeza-Yates, R., Boldi, P., Chierichetti, F.: Essential Web pages are easy to find. In: Proceedings of WWW, pp. 97–107 (2015)Google Scholar
  6. 6.
    Bagheri, E., Ensan, F.: Semantic tagging and linking of software engineering social content. Autom. Softw. Eng. 23(2), 147–190 (2016)Google Scholar
  7. 7.
    Balabanović, M., Shoham, Y.: Fab: content-based, collaborative recommendation. Commun. ACM 40(3), 66–72 (1997)CrossRefGoogle Scholar
  8. 8.
    Barragáns-Martínez, A.B., Costa-Montenegro, E., Burguillo, J.C., Rey-López, M., Mikic-Fonte, F.A., Peleteiro, A.: A hybrid content-based and item-based collaborative filtering approach to recommend tv programs enhanced with singular value decomposition. Inform. Sci. 180(22), 4290–4311 (2010)CrossRefGoogle Scholar
  9. 9.
    Bernstein, M.S., Teevan, J., Dumais, S., Liebling, D., Horvitz, E.: Direct answers for search queries in the long tail. In: Proceedings of SIGCHI, pp. 237–246 (2012)Google Scholar
  10. 10.
    Billsus, D., Pazzani, M.J.: User modeling for adaptive news access. User Model. User-Adap. Inter. 10(2-3), 147–180 (2000)CrossRefGoogle Scholar
  11. 11.
    Blanco, R., Cambazoglu, B.B., Mika, P., Torzec, N.: Entity recommendations in Web search. In: Proceedings of ISWC, pp. 33–48 (2013)Google Scholar
  12. 12.
    Bobadilla, J., Ortega, F., Hernando, A., Gutiérrez, A.: Recommender systems survey. Knowl.-Based Syst. 46, 109–132 (2013)CrossRefGoogle Scholar
  13. 13.
    Brandt, J., Guo, P.J., Lewenstein, J., Dontcheva, M., Klemmer, S.R.: Two studies of opportunistic programming: interleaving Web foraging, learning, and writing code. In: Proceedings of CHI, pp. 1589–1598 (2009)Google Scholar
  14. 14.
    Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of UAI, pp. 43–52 (1998)Google Scholar
  15. 15.
    Bretherton, I.: Attachment theory: retrospect and prospect. Monographs Soc. Res. Child Develop. 50(1/2), 3–35 (1985)Google Scholar
  16. 16.
    Broder, A.: A taxonomy of Web search. In: Proceedings of SIGIR, vol. 36, pp. 3–10 (2002)Google Scholar
  17. 17.
    Celma, Ò., Serra, X.: Foafing the music: bridging the semantic gap in music recommendation. Web Semant. Sci. Serv. Agents World Wide Web 6(4), 250–256 (2008)CrossRefGoogle Scholar
  18. 18.
    Chowdhury, G.G.: Introduction to Modern Information Retrieval. Facet Publishing, London (2010)Google Scholar
  19. 19.
    Cooley, R., Mobasher, B., Srivastava, J.: Web mining: information and pattern discovery on the World Wide Web. In: Proceedings of ICTAI, pp. 558–567 (1997)Google Scholar
  20. 20.
    Das, A.S., Datar, M., Garg, A., Rajaram, S.: Google news personalization: scalable online collaborative filtering. In: Proceedings of WWW, pp. 271–280 (2007)Google Scholar
  21. 21.
    Degemmis, M., Lops, P., Semeraro, G.: A content-collaborative recommender that exploits wordnet-based user profiles for neighborhood formation. Proceedings of UMAP 17(3), 217–255 (2007)Google Scholar
  22. 22.
    Dou, Z., Song, R., Nie, J.-Y., Wen, J.-R.: Using anchor texts with their hyperlink structure for Web search. In: Proceedings of SIGIR, pp. 227–234 (2009)Google Scholar
  23. 23.
    Gomez, C., Cleary, B., Singer, L.: A study of innovation diffusion through link sharing on stack overflow. In: Proceedings of MSR, pp. 81–84 (2013)Google Scholar
  24. 24.
    Gong, Y., Zhang, Q.: Hashtag recommendation using attention-based convolutional neural network. In: Proceedings of IJCAI, pp. 2782–2788. AAAI Press (2016)Google Scholar
  25. 25.
    Grbovic, M., Radosavljevic, V., Djuric, N., Bhamidipati, N., Savla, J., Bhagwan, V., Sharp, D.: E-commerce in your inbox: product recommendations at scale. In: Proceedings of KDD, pp. 1809–1818 (2015)Google Scholar
  26. 26.
    Guo, H., Tang, R., Ye, Y., Li, Z., Deepfm, X.H.E.: A factorization-machine based neural network for ctr prediction. In: Proceedings of IJCAI (2017)Google Scholar
  27. 27.
    He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.-S.: Neural collaborative filtering. In: Proceedings of WWW, pp. 173–182. International World Wide Web Conferences Steering Committee (2017)Google Scholar
  28. 28.
    Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. In: Proceedings of ICLR (2015)Google Scholar
  29. 29.
    Holmes, R., Walker, R.J., Murphy, G.C.: Strathcona example recommendation tool. In: Proceedings of Software Engineering Notes, vol. 30, pp. 237–240 (2005)Google Scholar
  30. 30.
    Karypis, G.: Evaluation of item-based top-N recommendation algorithms. In: Proceedings of CIKM, pp. 247–254. ACM, New York (2001)Google Scholar
  31. 31.
    Klemmer, S.R., Sinha, A.K., Chen, J., Landay, J.A., Aboobaker, N., Wang, A.: Suede: a wizard of oz prototyping tool for speech user interfaces. In: Proceedings of UIST, pp. 1–10. ACM (2000)Google Scholar
  32. 32.
    Krompaß, D., Baier, S., Tresp, V.: Type-constrained representation learning in knowledge graphs. In: Proceedings of ISWC, pp. 640–655. Springer (2015)Google Scholar
  33. 33.
    Levandoski, J.J., Sarwat, M., Eldawy, A., Mokbel, M.F.: Lars: a location-aware recommender system. In: Proceedings of ICDE, pp. 450–461 (2012)Google Scholar
  34. 34.
    Li, Y., Lu, L., Xuefeng, L.: A hybrid collaborative filtering method for multiple-interests and multiple-content recommendation in e-commerce. Expert Syst. Appl. 28(1), 67–77 (2005)CrossRefGoogle Scholar
  35. 35.
    Li, H., Zhao, X., Xing, Z., Bao, L., Peng, X., Gao, D., Zhao, W.: amassist: in-ide ambient search of online programming resources. In: Proceedings of SANER, pp. 390–398 (2015)Google Scholar
  36. 36.
    Li, J., Bao, L., Xing, Z., Wang, X., Zhou, B.: Bpminer: mining developers’ behavior patterns from screen-captured task videos. In: Proceedings of SAC, pp. 1371–1377. ACM (2016)Google Scholar
  37. 37.
    Li, J., Xing, Z., Ye, D., Zhao, X.: From discussion to wisdom: Web resource recommendation for hyperlinks in stack overflow. In: Proceedings of SAC, pp. 1127–1133. ACM (2016)Google Scholar
  38. 38.
    Li, J., Sun, A., Xing, Z.: Learning to answer programming questions with software documentation through social context embedding. Inform. Sci. 448-449, 36–52 (2018)CrossRefGoogle Scholar
  39. 39.
    Li, J., Sun, A., Xing, Z., Han, L.: Api caveat explorer: surfacing negative usages from practice. In: Proceedings of SIGIR, pp. 1293–1296. (2018)
  40. 40.
    Li, J., Xing, Z., Kabir, A.: Leveraging official content and social context to recommend software documentation. IEEE Transactions on Services Computing. IEEE Early Access (2018),
  41. 41.
    Lilliefors, H.W.: On the kolmogorov-smirnov test for normality with mean and variance unknown. J. Am. Stat. Assoc. 62(318), 399–402 (1967)CrossRefGoogle Scholar
  42. 42.
    Lin, T., Pantel, P., Gamon, M., Kannan, A., Fuxman, A.: Active objects: actions for entity-centric search. In: Proceedings of WWW, pp. 589–598 (2012)Google Scholar
  43. 43.
    Linden, G., Smith, B., York, J.: recommendations: item-to-item collaborative filtering. IEEE Internet Comput. 7(1), 76–80 (2003)CrossRefGoogle Scholar
  44. 44.
    Liu, T.-Y.: Learning to rank for information retrieval. Found. Trend Inform. Retrieval 3(3), 225–331 (2009)CrossRefGoogle Scholar
  45. 45.
    Luo, X., Xia, Y., Zhu, Q.: Incremental collaborative filtering recommender based on regularized matrix factorization. Know.-Based Syst. 27, 271–280 (2012)CrossRefGoogle Scholar
  46. 46.
    Magnini, B., Strapparava, C.: Improving user modelling with content-based techniques. In: Proceedings of the International Conference on User Modeling, pp. 74–83. Springer (2001)Google Scholar
  47. 47.
    Marchionini, G.: Exploratory search: from finding to understanding. Commun. ACM 49(4), 41–46 (2006)CrossRefGoogle Scholar
  48. 48.
    McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Proceedings of SIGIR, pp. 329–336. ACM (2004)Google Scholar
  49. 49.
    Melville, P., Mooney, R.J., Nagarajan, R.: Content-boosted collaborative filtering for improved recommendations. In: Eighteenth National Conference on Artificial Intelligence, pp. 187–192. American Association for Artificial Intelligence, Menlo Park (2002)Google Scholar
  50. 50.
    Miliaraki, I., Blanco, R., Lalmas, M.: From Selena Gomez to Marlon Brando: understanding explorative entity search. In: Proceedings of WWW, pp. 765–775 (2015)Google Scholar
  51. 51.
    Mooney, R.J., Roy, L.: Content-based book recommending using learning for text categorization. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 195–204. ACM (2000)Google Scholar
  52. 52.
    Nguyen, P.T., Tomeo, P., Di Noia, T., Di Sciascio, E.: Content-based recommendations via dbpedia and freebase: a case study in the music domain. In: Proceedings of ISWC, pp. 605–621. Springer (2015)Google Scholar
  53. 53.
    Nicholas, I.S.C., Nicholas, C.K.: Combining content and collaboration in text filtering. In: Proceedings of IJCAI, pp. 86–91 (1999)Google Scholar
  54. 54.
    Nilashi, M., Ibrahim, O.B., Ithnin, N., Zakaria, R.: A multi-criteria recommendation system using dimensionality reduction and neuro-fuzzy techniques. Soft. Comput. 19(11), 3173–3207 (2015)CrossRefGoogle Scholar
  55. 55.
    Nilashi, M., Ibrahim, O., Bagherifard, K.: A recommender system based on collaborative filtering using ontology and dimensionality reduction techniques. Expert Syst. Appl. 92, 507–520 (2018)CrossRefGoogle Scholar
  56. 56.
    Park, M.-H., Hong, J.-H., Cho, S.-B.: Location-based recommendation system using Bayesian user’s preference model in mobile devices. In: Ubiquitous Intelligence and Computing, pp. 1130–1139. Springer (2007)Google Scholar
  57. 57.
    Park, S.-T., Pennock, D.M.: Applying collaborative filtering techniques to movie search for better ranking and browsing. In: Proceedings of KDD, pp. 550–559 (2007)Google Scholar
  58. 58.
    Parnin, C., Treude, C., Grammel, L., Storey, M.-A.: Crowd documentation: exploring the Coverage and the Dynamics of Api Discussions on Stack Overflow. Georgia Institute of Technology, Tech. Rep (2012)Google Scholar
  59. 59.
    Pazzani, M., Billsus, D.: Learning and revising user profiles: the identification of interesting Web sites. Mach. Learn. 27(3), 313–331 (1997)CrossRefGoogle Scholar
  60. 60.
    Ponzanelli, L., Bacchelli, A., Lanza. M.: Seahawk: stack overflow in the ide. In: Proceedings of ICSE, pp. 1295–1298 (2013)Google Scholar
  61. 61.
    Ponzanelli, L., Bavota, G., Di Penta, M., Oliveto, R., Lanza, M.: Prompter: a self-confident recommender system. In: Proceedings of ICSME, pp. 577–580 (2014)Google Scholar
  62. 62.
    Sakai, T., Nogami, K.: Serendipitous search via wikipedia: a query log analysis. In: Proceedings of SIGIR, pp. 780–781 (2009)Google Scholar
  63. 63.
    Salakhutdinov, R., Mnih, A., Hinton, G.: Restricted Boltzmann machines for collaborative filtering. In: Proceedings of ICML, pp. 791–798. ACM, New York (2007)Google Scholar
  64. 64.
    San Pedro, J., Karatzoglou, A.: Question recommendation for collaborative question answering systems with rankslda. In: Proceedings of RecSys, pp. 193–200 (2014)Google Scholar
  65. 65.
    Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of WWW, pp. 285–295 (2001)Google Scholar
  66. 66.
    Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: Methods and metrics for cold-start recommendations. In: Proceedings of SIGIR, pp. 253–260. ACM, New York (2002)Google Scholar
  67. 67.
    Sedhain, S., Menon, A.K., Sanner, S., Xie, L.: Autorec: autoencoders meet collaborative filtering. In: Proceedings of WWW, pp. 111–112. ACM (2015)Google Scholar
  68. 68.
    Seebach, C.: Searching for answers–knowledge exchange through social media in organizations. In: Proceedings of HICSS, pp. 3908–3917 (2012)Google Scholar
  69. 69.
    Sheth, B., Maes, P.: Evolving agents for personalized information filtering. In: Proceedings of the Ninth Conference on Artificial Intelligence for Applications, pp. 345–352. IEEE (1993)Google Scholar
  70. 70.
    Shinde, S.K., Kulkarni, U.: Hybrid personalized recommender system using centering-bunching based clustering algorithm. Expert Syst. Appl. 39(1), 1381–1387 (2012)CrossRefGoogle Scholar
  71. 71.
    Singhal, A., et al.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)Google Scholar
  72. 72.
    Su, X., Khoshgoftaar, T.M.: A survey of collaborative filtering techniques. Advan. Artif. Intell. 2009, 4 (2009)Google Scholar
  73. 73.
    Subramanian, S., Inozemtseva, L., Holmes, R.: Live api documentation. In: Proceedings of ICSE, pp. 643–652 (2014)Google Scholar
  74. 74.
    Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., Qin, B.: Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of ACL, vol. 1, pp. 1555–1565 (2014)Google Scholar
  75. 75.
    Tenopir, C., King, D.W.: Communication Patterns of Engineers. Wiley, New York (2004)Google Scholar
  76. 76.
    Tran, T., Phung, D., Venkatesh, S.: Collaborative filtering via sparse markov random fields. Inform. Sci. 369, 221–237 (2016)CrossRefGoogle Scholar
  77. 77.
    Treude, C., Barzilay, O., Storey, M.-A.: How do programmers ask and answer questions on the Web?: Nier track. In: Proceedings of ICSE, pp. 804–807 (2011)Google Scholar
  78. 78.
    Treude, C., Robillard, M.: Augmenting api documentation with insights from stack overflow. In: Proceedings of ICSEGoogle Scholar
  79. 79.
    Wang, S., Lo, D., Jiang, L.: An empirical study on developer interactions in stackoverflow. In: Proceedings of SAC, pp. 1019–1024 (2013)Google Scholar
  80. 80.
    Wang, S., Lo, D., Vasilescu, B., Serebrenik, A.: Entagrec: an enhanced tag recommendation system for software information sites. In: Proceedings of ICSME, pp. 291–300 (2014)Google Scholar
  81. 81.
    Wang, F.-H., Jian, S.-Y.: An effective content-based recommendation method for Web browsing based on keyword context matching. J. Inform. Electron. 1(2), 49–59 (2006)MathSciNetGoogle Scholar
  82. 82.
    Wang, J., De Vries, A.P., Reinders, M.J.: Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In: Proceedings of SIGIR, pp. 501–508 (2006)Google Scholar
  83. 83.
    Wang, J., Yu, L., Zhang, W., Gong, Y., Xu, Y., Wang, B., Zhang, P., Zhang, D.: Irgan: a minimax game for unifying generative and discriminative information retrieval models. In: Proceedings of SIGIR, pp. 515–524. ACM (2017)Google Scholar
  84. 84.
    White, R.W., Roth, R.A.: Exploratory search: beyond the query-response paradigm. Synth. Lect. Inform. Concepts Retr. Serv. 1(1), 1–98 (2009)Google Scholar
  85. 85.
    Yager, R.R.: Fuzzy logic methods in recommender systems. Fuzzy Sets Syst. 136 (2), 133–149 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  86. 86.
    Ye, M., Yin, P., Lee, W.-C., Lee, D.-L.: Exploiting geographical influence for collaborative point-of-interest recommendation. In: Proceedings of SIGIR, pp. 325–334. ACM, New York (2011)Google Scholar
  87. 87.
    Ye, D., Xing, Z., Foo, C.Y., Ang, Z.Q., Li, J., Kapre, N.: Software-specific named entity recognition in software engineering social content. In: Proceedings of SANER, vol. 1, pp. 90–101. IEEE (2016)Google Scholar
  88. 88.
    Ye, D., Xing, Z., Li, J., Kapre, N.: Software-specific part-of-speech tagging: an experimental study on stack overflow. In: Proceedings of SAC, pp. 1378–1385. ACM (2016)Google Scholar
  89. 89.
    Yu, X., Ma, H., Hsu, B.-J.P., Han, J.: On building entity recommender systems using user click log and freebase knowledge. In: Proceedings of WSDM, pp. 263–272. ACM (2014)Google Scholar
  90. 90.
    Yuan, Q., Cong, G., Sun, A.: Graph-based point-of-interest recommendation with geographical and temporal influences. In: Proceedings of CIKM, pp. 659–668 (2014)Google Scholar
  91. 91.
    Zagalsky, A., Barzilay, O., Yehudai, A.: Example overflow: using social media for code recommendation. In: Proceedings of the Third International Workshop on Recommendation Systems for Software Engineering, pp. 38–42 (2012)Google Scholar
  92. 92.
    Zhang, S., Yao, L., Sun, A.: Deep learning based recommender system: a survey and new perspectives. arXiv:1707.07435 (2017)
  93. 93.
    Zhou, G., He, T., Zhao, J., Hu, P.: Learning continuous word embedding with metadata for question retrieval in community question answering. In: Proceedings of ACL, pp. 250–259 (2015)Google Scholar
  94. 94.
    Zimmermann, T., Dallmeier, V., Halachev, K., Zeller, A.: erose: guiding programmers in eclipse. In: Proceedings of SPLASH, pp. 186–187 (2005)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer Science and EngineeringNanyang Technological UniversitySingaporeSingapore
  2. 2.College of Engineering and Computer ScienceAustralian National UniversityCanberraAustralia

Personalised recommendations