Building Web-Scale Data Mining Infrastructure for Search

Abstract

The competition in search has been driving great innovation and investment on next generation Internet services, with the goal of providing the computing platform for Internet economies on a global scale. Different from traditional Internet services, search involves myriad offline computations to analyze data at a very large scale, and an infrastructure for “scale” experiments is often required to evaluate the effectiveness of newly invented algorithms in a simulated “real” environment. In this talk, I will first review a variety of new trends in computational economies on the Internet in which search and online advertising have become the driving forces in building underlying computing infrastructure. Then, I will introduce current efforts at Microsoft Research Asia on building this new infrastructure. I will also discuss how these efforts are influencing the design of next-generation search engines from an architecture stand-point. Some advanced search technologies based on the use of this infrastructure and deeper data mining on the Web will also be demonstrated.