Crawling and Detecting Community Structure in Online Social Networks Using Local Information

  • Norbert Blenn
  • Christian Doerr
  • Bas Van Kester
  • Piet Van Mieghem
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7289)


As Online Social Networks (OSNs) become an intensive subject of research for example in computer science, networking, social sciences etc., a growing need for valid and useful datasets is present. The time taken to crawl the network is however introducing a bias which should be minimized. Usual ways of addressing this problem are sampling based on the nodes (users) ids in the network or crawling the network until one “feels” a sufficient amount of data has been obtained.

In this paper we introduce a new way of directing the crawling procedure to selectively obtain communities of the network. Thus, a researcher is able to obtain those users belonging to the same community and rapidly begin with the evaluation. As all users involved in the same community are crawled first, the bias introduced by the time taken to crawl the network and the evolution of the network itself is less.

Our presented technique is also detecting communities during runtime. We compare our method called Mutual Friend Crawling (MFC) to the standard methods Breadth First Search (BFS) and Depth First Search (DFS) and different community detection algorithms. The presented results are very promising as our method takes only linear runtime but is detecting equal structures as modularity based community detection algorithms.


Social Networks Community Detection Crawling 


  1. 1.
    Clauset, A., Newman, M.E.J., Moore, C.: Finding community structure in very large networks. Physical Review E 70 (December 2004)Google Scholar
  2. 2.
    Trajanovski, S., Wang, H., Van Mieghem, P.: Maximum Modular Graphs. European Physics Journal B (2011) (submitted)Google Scholar
  3. 3.
    Cormen, T.: Introduction to algorithms. MIT electrical engineering and computer science series. MIT Press (2001)Google Scholar
  4. 4.
    Feld, S.L.: Why Your Friends Have More Friends Than You Do. American Journal of Sociology 96(6), 1464–1477 (1991)CrossRefGoogle Scholar
  5. 5.
    Kurant, M., Markopoulou, A., Thiran, P.: On the bias of BFS (Breadth First Search). In: 22nd International Teletraffic Congress (ITC), pp. 1–8. IEEE (2010)Google Scholar
  6. 6.
    Pons, P., Latapy, M.: Computing Communities in Large Networks Using Random Walks. In: Yolum, p., Güngör, T., Gürgen, F., Özturan, C. (eds.) ISCIS 2005. LNCS, vol. 3733, pp. 284–293. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  7. 7.
    Lai, D., Lu, H., Nardini, C.: Enhanced modularity-based community detection by random walk network preprocessing. Phys. Rev. E 81, 066118 (2010)CrossRefGoogle Scholar
  8. 8.
    Reichardt, J., Bornholdt, S.: Statistical mechanics of community detection. Phys. Rev. E Stat. Nonlin. Soft. Matter Phys. 74 (July 2006)Google Scholar
  9. 9.
    Girvan, M., Newman, M.E.J.: Community structure in social and biological networks. Proceedings of the National Academy of Sciences 99, 7821–7826 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Newman, M.E.J., Girvan, M.: Finding and evaluating community structure in networks. Phys. Rev. E 69, 026113 (2004)CrossRefGoogle Scholar
  11. 11.
    Raghavan, U.N., Albert, R., Kumara, S.: Near linear time algorithm to detect community structures in large-scale networks. Physical Review E 76, 036106+ (2007)Google Scholar
  12. 12.
    Nguyen, N., Dinh, T., Xuan, Y., Thai, M.: Adaptive algorithms for detecting community structure in dynamic social networks. In: 2011 Proceedings IEEE INFOCOM, pp. 2282–2290 (April 2011)Google Scholar
  13. 13.
    Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of community hierarchies in large networks. CoRR, abs/0803.0476 (2008)Google Scholar
  14. 14.
    Fortunato, S., Castellano, C.: Community structure in graphs (2007)Google Scholar
  15. 15.
    Van Kester, S.: Efficient Crawling of Community Structures in Online Social Networks. PVM 2011-071, Tu Delft (September 2011)Google Scholar
  16. 16.
    Zachary, W.W.: An Information Flow Model for Conflict and Fission in Small Groups. Journal of Anthropological Research 33(4) (1977)Google Scholar
  17. 17.
    Tang, S., Blenn, N., Doerr, C., Van Mieghem, P.: Digging in the Digg Social News Website. IEEE Transactions on Multimedia 13, 1163–1175 (2011)CrossRefGoogle Scholar
  18. 18.
    Fortunato, S., Barthélemy, M.: Resolution limit in community detection. Proceedings of the National Academy of Sciences 104, 36–41 (2007)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  • Norbert Blenn
    • 1
  • Christian Doerr
    • 1
  • Bas Van Kester
    • 1
  • Piet Van Mieghem
    • 1
  1. 1.Department of TelecommunicationTU DelftDelftThe Netherlands

Personalised recommendations