Using Text for Prediction

Abstract

Once text is transformed into numerical vectors, automated prediction methods can be applied. Predictive text mining is described in terms of an empirical analysis that looks for word patterns, especially for document classification. Fundamental methods of machine learning from sample data are outlined including similarity-based methods, decision rules and trees, probabilistic methods and linear methods. Evaluation techniques are examined to estimate future performance and to maximize empirical results. Errors and pitfalls in big data evaluation are considered, and graph models for social networks are introduced.