Journal of Central South University of Technology

, Volume 16, Issue 6, pp 976–981 | Cite as

Log integration on large scale for global networking monitoring

  • Jia-jia Miao (缪嘉嘉)Email author
  • Quan-yuan Wu (吴泉源)
  • Yan Jia (贾 焰)


Supposing that the overall situation is dug out from the distributed monitoring nodes, there should be two critical obstacles, heterogenous schema and instance, to integrating heterogeneous data from different monitoring sensors. To tackle the challenge of heterogenous schema, an instance-based approach for schema mapping, named instance-based machine-learning (IML) approach was described. And to solve the problem of heterogenous instance, a novel approach, called statistic-based clustering (SBC) approach, which utilized clustering and statistics technologies to match large scale sources holistically, was also proposed. These two algorithms utilized the machine-leaning and clustering technology to improve the accuracy. Experimental analysis shows that the IML approach is more precise than SBC approach, reaching at least precision of 81% and recall rate of 82%. Simulation studies further show that SBC can tackle large scale sources holistically with 85% recall rate when there are 38 data sources.

Key words

machine-learning clustering data integration schema matching instance matching 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    US-CERT. Technical cyber security alerts[EB/OL]. [2005-10-04].
  2. [2]
    LI Xiang, LIU Guang-ying, QI Jian-xun. Fuzzy neural and chaotic searching hybrid algorithm and its application in electric customers’s credit risk evaluation[J]. Journal of Central South University of Technology, 2007, 14(1): 140–143.CrossRefGoogle Scholar
  3. [3]
    MIAO Jia-jia. GS-TMS: A global stream-based threat monitor system[C]//Proceedings of the 34th International Conference on Very Large Data Bases. Auckland: VLDB Endowment, 2008: 1678–1687.Google Scholar
  4. [4]
    DOAN A, HALEVY A Y. Semantic-integration research in the database community[J]. AI Magazine, 2005, 26(5): 183–194.Google Scholar
  5. [5]
    RAHM E, BERNSTEIN P A. A survey of approaches to automatic schema matching[J]. The VLDB Journal, 2001, 11(1): 334–350.CrossRefGoogle Scholar
  6. [6]
    LI W, CLIFTON C. SEMINT: A tool for identifying attribute correspondences in heterogeneous databases using neural networks[J]. Data and Knowledge Engineering, 2000, 3(4): 49–84.CrossRefGoogle Scholar
  7. [7]
    DOAN A, DOMINGOS P, HALEVY A Y. Reconciling schemas of disparate data sources: A machine-learning approach[C]//Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data. Santa Barbara: ACM Press, 2001: 509–520.CrossRefGoogle Scholar
  8. [8]
    HAAS L M. Clio grows up: From research prototype to industrial tool[C]//Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data. Baltimore: ACM Press, 2005: 805–810.CrossRefGoogle Scholar
  9. [9]
    MILLER R J. The Clio project: Managing heterogeneity[J]. SIGMOD Record, 2001, 30: 78–83.CrossRefGoogle Scholar
  10. [10]
    HE B, CHANG K C. Making holistic schema matching robust: An ensemble approach[C]//Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining. Chicago: ACM Press, 2005: 429–438.Google Scholar
  11. [11]
    HE B, CHANG K C. Discovering complex matching across web query interfaces: A correlation mining approach[C]//Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Seattle: ACM Press, 2004: 148–157.Google Scholar
  12. [12]
    YANG Q, ZHANG H H, LI T. Mining web logs for prediction models in www caching and perfecting[C]//Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Chicago: ACM Press, 2001: 473–478.Google Scholar
  13. [13]
    TING K M, WITTEN I H. Issues in stacked generalization[J]. Journal of Artif Intell Res, 1999, 10(5): 271–289.CrossRefGoogle Scholar
  14. [14]
    DAEMI A, CALMET J. From ontologies to trust through entropy[C]//Proceedings of the International Conference on Advances in Intelligent Systems — Theory and Applications. Luxembourg: IEEE Computer Society, 2004: 12–43.Google Scholar
  15. [15]
    BERKHIN P. A survey of clustering data mining techniques[J]. Grouping Multidimensional Data, 2006, 1(2): 25–71.CrossRefGoogle Scholar

Copyright information

© Central South University Press and Springer Berlin Heidelberg 2009

Authors and Affiliations

  • Jia-jia Miao (缪嘉嘉)
    • 1
    • 2
    Email author
  • Quan-yuan Wu (吴泉源)
    • 1
  • Yan Jia (贾 焰)
    • 1
  1. 1.School of ComputerNational University of Defense TechnologyChangshaChina
  2. 2.Institute of Command AutomationPLA University of Science and TechnologyNanjingChina

Personalised recommendations