Modelability Criteria: Statistical Characteristics Estimating Feasibility to Build Predictive QSAR Models for a Dataset

Abstract

It is not always possible to build predictive Quantitative Structure-Activity Relationships (QSAR) models for a given chemical dataset. In this work, we propose several statistical criteria, which can with high confidence answer a question, whether it is possible to build a predictive model for a dataset prior to actual modeling, i.e. to establish, whether the dataset is modelable. Calculation of these criteria is fast, and using them in QSAR studies could dramatically reduce modelers’ time and efforts, as well as computational resources necessary to build QSAR models for at least some datasets, especially for those which are not modelable. The calculation of modelability criteria is based on the k-nearest neighbors approach. For all datasets, as modelability criteria we have proposed dataset diversity (MODI_DIV) and new activity cliff indices (MODI_ACI). For datasets with binary end points, as modelability criteria we have proposed the correct classification rate (MODI_CCR) CCR = 0.5(sensitivity + specificity) for leave-one-out (LOO) cross-validation in the entire descriptor space, and correct classification rate for similarity search (MODI_ssCCR) in the entire descriptor space with leave 20 %-out (five-fold) cross-validation. For binary datasets, all these modelability criteria were tested on 42 datasets with previously generated QSAR models. Two latter criteria (MODI_CCR and MODI_ssCCR) were found to have high correlation with the predictivity of QSAR models (QSAR_CCR) and were additionally tested on 60 ToxCast end points with QSAR modeling results published recently (Thomas RS, Black MB, Li L, Healy E, Chu T-M, Bao W, Andersen MD, Wolfinger RD. Toxicol Sci: Off JSoc Toxicol 128(2):398–417, 2012). These modelability criteria can be used to classify many datasets as modelable or non-modelable. These criteria can be generalized to datasets with compounds belonging to more than two categories or classes. Additionally, criteria which take into account errors of prediction MODI_CAT _i and MODI_CLASS _i were proposed for datasets with compounds belonging to more than two (i > 2) categories or classes and continuous end points, divided into i > 2 bins. For continuous end points, LOO cross-validation q ² for similarity search with different numbers of nearest neighbors in the entire descriptor space (MODI_q ²), and similarity search coefficient of determination (MODI_ssR ²) in the entire descriptor space were proposed as modelability criteria. Our preliminary studies demonstrated high correlation between the external predictivity of QSAR models (QSAR_R ²) and each of the MODI_q ² and MODI_ssR ². On the other hand, for datasets with any binary or continuous response variable, MODI_DIVs and MODI_ACIs were found to be less useful to establish dataset modelability.