Text Mining: The New Data Mining Frontier

Abstract

Data miners and analysts are used to dealing with data in structured fields, such as rows and columns. Data, such as age, income, purchase spending, and purchase dates, are typically found in specific columns or fields. Advances in data mining technology now allow analysis of unstructured text data. In other words, information pertaining to language and communication can now be analyzed. This emerging discipline is commonly referred to as text mining or text analytics. One question that may immediately come to mind is how this differs from search engine technology. In search engine technology, users list key words or phrases that are then analyzed to produce a list of articles, themes, or topics supposedly related to the key words. Text mining also analyzes text, but users do not enter specific key words or phrases; rather, users do not know what they are looking for. A body of text or unstructured data, called a corpus, is presented to the analyst. With text mining tools the analyst then identifies patterns or themes in this corpus. For example, XYZ company may want to use some market research analysis of its customer base to better understand why the number of complaints has increased. Text mining could be employed here to extract all the customer e-mails of the past several months. By analyzing all these emails, the analyst may uncover themes and/ or discussion patterns that could help identify specific reasons for the increasing customer complaints.