Cobweb/IDX: Mapping Cobweb to SQL

Data-mining algorithms are used in many applications to help extract meaningful data from very large datasets. For example, the NetFlix [12] Web site uses hundreds of thousands of past movie ratings stored in an Oracle database to propose movies to returning customers.

Existing data-mining algorithms extract data from databases before processing them but this requires a lot of time and expertise from database administrators. One method of simplifying this process is to develop the algorithms as part of the database management system (DBMS) and to make them accessible using standard database querying tools. However, there are many challenges to be overcome before data mining can be performed using off-the-shelf query tools. One challenge is to make the process of asking a question and interpreting the results as simple as querying a database table. A second challenge is to develop data-mining algorithms that use the database efficiently because database access can have major performance implications.

This chapter suggests one solution to the challenge of making the data-mining process simpler. It discusses an implementation of a popular conceptual clustering algorithm, Cobweb [4], as an add-on to a DBMS. We call our implementation Cobweb/IDX. Section 26.2 is a discussion of the Cobweb algorithm. Section 26.3 discusses the motivation for choosing Cobweb as the basis for our work. Section 26.4 discusses Cobweb/IDX and how it maps the Cobweb algorithm to SQL. Section 26.5 talks about the advantages and disadvantages of the Cobweb/IDX implementation. Section 26.6 presents other work on integrating data mining with databases and finally Sect. 26.7 contains a summary and directions for future work.