Hybrid Clustering Based Smart Crawler

Abstract

Everyone can easily use the Internet because it is the source of information. The Internet holds a tremendous amount of information, get the relevant information and high efficiency is challenging issue. The crawler is helping to extract relevant information easily. We propose smart crawler which provides better results than another crawler. Smart crawler has two stages as site locating and in-site exploring. Site locating can fetch the relevant information and in-site exploring rank the sites as per their relevancy. We implement smart crawler using a hybrid algorithm. The hybrid algorithm is a combination of bisect K-means algorithm and bottom up agglomerative clustering algorithm. The bisect K-means clustering algorithm split cluster into sub clusters and agglomerative algorithm computes the centroid of the clusters and merge them on the basis of similarities or approximate similar centroid value and form singleton cluster hence it is simple to use and provides efficient and better result than K-means clustering algorithm as it generates uniform size clusters.