The randomized information coefficient: assessing dependencies in noisy data
When differentiating between strong and weak relationships using information theoretic measures, the variance plays an important role: the higher the variance, the lower the chance to correctly rank the relationships. We propose the randomized information coefficient (RIC), a mutual information based measure with low variance, to quantify the dependency between two sets of numerical variables. We first formally establish the importance of achieving low variance when comparing relationships using the mutual information estimated with grids. Second, we experimentally demonstrate the effectiveness of RIC for (i) detecting noisy dependencies and (ii) ranking dependencies for the applications of genetic network inference and feature selection for regression. Across these tasks, RIC is very competitive over other 16 state-of-the-art measures. Other prominent features of RIC include its simplicity and efficiency, making it a promising new method for dependency assessment.
KeywordsDependency measures Noisy relationships Normalized mutual information Randomized ensembles
Simone Romano’s work was supported by a Melbourne International Research Scholarship (MIRS). James Bailey’s work was supported by an Australian Research Council Future Fellowship. Experiments were carried out on Amazon cloud supported by AWS in Education Grant Award.
- Dougherty, J., Kohavi, R., Sahami, M., et al. (1995). Supervised and unsupervised discretization of continuous features. Machine learning: Proceedings of the twelfth international conference, 12, 194–202.Google Scholar
- Faivishevsky, L. & Goldberger, J. (2009). ICA based on a smooth estimation of the differential entropy. In Advances in neural information processing systems (pp. 433–440).Google Scholar
- Fayyad, U. & Irani, K. (1993). Multi-interval discretization of continuous-valued attributes for classification learning. In International joint conference on artificial intelligence (IJCAI) Google Scholar
- Geurts, P. (2002). Bias/variance tradeoff and time series classification. PhD thesis, Department d’Életrecité, Életronique et Informatique. Institut Momntefiore. Unversité de Liège.Google Scholar
- Gretton, A., Bousquet, O., Smola, A., & Schölkopf, B. (2005). Measuring statistical dependence with Hilbert–Schmidt norms. In Algorithmic learning theory (pp. 63–77). Springer.Google Scholar
- Huo, X. & Szekely, G. J. (2014). Fast computing for distance covariance. ArXiv preprint arXiv:1410.1503.
- Khan, S., Bandyopadhyay, S., Ganguly, A. R., Saigal, S., Erickson, D. J., I. I. I., Protopopescu, V., et al. (2007). Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E, 76(2), 026209.Google Scholar
- Kraskov, A., Stögbauer, H., & Grassberger, P. (2004). Estimating mutual information. Physical Review E, 69(6), 066138.Google Scholar
- Lippert, C., Stegle, O., Ghahramani, Z., & Borgwardt, K. M. (2009). A kernel method for unsupervised structured network inference. In International conference on artificial intelligence and statistics (pp. 368–375).Google Scholar
- Lizier, J. T. (2014). JIDT: An information-theoretic toolkit for studying the dynamics of complex systems. ArXiv preprint arXiv:1408.3270.
- Lopez-Paz, D., Hennig, P., & Schölkopf, B. (2013). The randomized dependence coefficient. In Advances in neural information processing systems (pp. 1–9).Google Scholar
- Luedtke, A. & Tran, L. (2013). The generalized mean information coefficient. ArXiv preprint arXiv:1308.5712.
- Nguyen, H. V., Müller, E., Vreeken, J., Efros, P., & Böhm, K. (2014a). Multivariate maximal correlation analysis. In Proceedings of the 31st international conference on machine learning (ICML-14) (pp. 775–783).Google Scholar
- Nguyen, H. V. & Vreeken, J. (2015). Universal dependency analysis. ArXiv preprint arXiv:1510.08389.
- Nguyen, X. V., Chan, J., Romano, S., & Bailey, J. (2014b). Effective global approaches for mutual information based feature selection. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 512–521). ACM.Google Scholar
- Özuysal, M., Fua, P., & Lepetit, V. (2007). Fast keypoint recognition in ten lines of code. In CVPR.Google Scholar
- Reshef, D. N., Reshef, Y. A., Sabeti, P. C., & Mitzenmacher, M. M. (2015a). An empirical study of leading measures of dependence. ArXiv preprint arXiv:1505.02214.
- Reshef, Y. A., Reshef, D. N., Finucane, H. K., Sabeti, P. C., & Mitzenmacher, M, M. (2015b). Measuring dependence powerfully and equitably. ArXiv preprint arXiv:1505.02213.
- Romano, S., Bailey, J., Nguyen, V., & Verspoor, K. (2014). Standardized mutual information for clustering comparisons: One step further in adjustment for chance. In Proceedings of the 31st international conference on machine learning (ICML-14) (pp. 1143–1151).Google Scholar
- Romano, S., Vinh, N. X., Bailey, J., & Verspoor, K. (2016). A framework to adjust dependency measure estimates for chance. In Proceedings of the 2016 SIAM international conference on data mining (pp. 423–431). Society for Industrial and Applied Mathematics.Google Scholar
- Schaffernicht, E., Kaltenhaeuser, R., Verma, S. S., & Gross, H.-M. (2010). On estimating mutual information for feature selection. In Artificial neural networks ICANN 2010 (pp. 362–367). Springer.Google Scholar
- Simon, N. & Tibshirani, R. (2011). Comment on detecting novel associations in large data sets. ArXiv preprint arXiv:1401.7645.
- Song, L., Smola, A., Gretton, A., Borgwardt, K. M., & Bedo, J. (2007). Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning (pp. 823–830). ACM.Google Scholar
- Sugiyama, M. & Borgwardt, K. M. (2013). Measuring statistical dependence via the mutual information dimension. In Proceedings of the twenty-third international joint conference on artificial intelligence (pp. 1692–1698). AAAI Press.Google Scholar
- Tang, D., Wang, M., Zheng, W., & Wang, H. (2014). Rapidmic: Rapid computation of the maximal information coefficient. Evolutionary Bioinformatics Online, 10, 11.Google Scholar
- Wang, Y., Romano, S., Nguyen, V., Bailey, J., Ma, X., & Xia, S.-T. (2017). Unbiased multivariate correlation analysis.Google Scholar