Statistics and Computing

, Volume 29, Issue 1, pp 43–51 | Cite as

Deep Gaussian mixture models

  • Cinzia ViroliEmail author
  • Geoffrey J. McLachlan


Deep learning is a hierarchical inference method formed by subsequent multiple layers of learning able to more efficiently describe complex relationships. In this work, deep Gaussian mixture models (DGMM) are introduced and discussed. A DGMM is a network of multiple layers of latent variables, where, at each layer, the variables follow a mixture of Gaussian distributions. Thus, the deep mixture model consists of a set of nested mixtures of linear models, which globally provide a nonlinear model able to describe the data in a very flexible way. In order to avoid overparameterized solutions, dimension reduction by factor models can be applied at each layer of the architecture, thus resulting in deep mixtures of factor analyzers.


Unsupervised classification Mixtures of factor analyzers Stochastic EM algorithm 


  1. Baek, J., McLachlan, G., Flack, L.: Mixtures of factor analyzers with common factor loadings: applications to the clustering and visualization of high-dimensional data. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1298–1309 (2010)CrossRefGoogle Scholar
  2. Baudry, J.-P., Raftery, A.E., Celeux, G., Lo, K., Gottardo, R.: Combining mixture components for clustering. J. Comput. Gr. Stat. 19(2), 332–353 (2010)MathSciNetCrossRefGoogle Scholar
  3. Celeux, G., Diebolt, J.: The SEM algorithm: a probabilistic teacher algorithm derived from the EM algorithm for the mixture problem. Comput. Stat. Q. 2(1), 73–82 (1985)Google Scholar
  4. Forina, M., Armanino, C., Castino, M., Ubigli, M.: Multivariate data analysis as a discriminating method of the origin of wines. Vitis 25(3), 189–201 (1986)Google Scholar
  5. Forina, M., Tiscornia, E.: Pattern-recognition methods in the prediction of Italian olive oil origin by their fatty-acid content. Anal. Chim. 72(3–4), 143–155 (1982)Google Scholar
  6. Fraley, C., Raftery, A.: Model-based clustering, discriminant analysis and density estimation. J. Am. Stat. Assoc. 97, 611–631 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  7. Hennig, C.: Methods for merging gaussian mixture components. Adv. Data Anal. Classif. 4(1), 3–34 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  9. Li, J.: Clustering based on a multilayer mixture model. J. Comput. Gr. Stat. 14(3), 547–568 (2005)MathSciNetCrossRefGoogle Scholar
  10. Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Academic Press, Oxford (1976)zbMATHGoogle Scholar
  11. McLachlan, G., Peel, D., Bean, R.: Modelling high-dimensional data by mixtures of factor analyzers. Comput. Stat. Data Anal. 41(3), 379–388 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  12. McLachlan, G.J., Peel, D.: Finite Mixture Models. Wiley, Hoboken (2000)CrossRefzbMATHGoogle Scholar
  13. Melnykov, V.: Merging mixture components for clustering through pairwise overlap. J. Comput. Gr. Stat. 25(1), 66–90 (2016)MathSciNetCrossRefGoogle Scholar
  14. Montanari, A., Viroli, C.: Heteroscedastic factor mixture analysis. Stat. Model. 10(4), 441–460 (2010)MathSciNetCrossRefGoogle Scholar
  15. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  16. Scrucca, L., Fop, M., Murphy, T.B., Raftery, A.E.: mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. R Journal 8, 289–317 (2016)Google Scholar
  17. Tang, Y., Hinton, G.E., Salakhutdinov, R.: Deep mixtures of factor analysers. In Langford, J., Pineau, J. (eds.) Proceedings of the 29th International Conference on Machine Learning (ICML-12), New York, NY, USA, pp. 505–512. ACM (2012)Google Scholar
  18. Taigman, Y., Yang, M., Ranzato, M.A., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701–1708 (2014)Google Scholar
  19. van den Oord, A., Schrauwen, B.: Factoring variations in natural images with deep gaussian mixture models. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 27, pp. 3518–3526. Curran Associates, Inc, Montreal, Quebec, Canada (2014)Google Scholar
  20. Viroli, C.: Dimensionally reduced model-based clustering through mixtures of factor mixture analyzers. J. Classif. 27(3), 363–388 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  21. Wang, K., Ng, S.-K., McLachlan, G.J.: Multivariate skew t mixture models: applications to fluorescence-activated cell sorting data. In: Digital Image Computing: Techniques and Applications, 2009. DICTA’09., pp. 526–531. IEEE (2009)Google Scholar
  22. Wei, G.C., Tanner, M.A.: A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. J. Am. Stat. Assoc. 85(411), 699–704 (1990)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of Statistical SciencesUniversity of BolognaBolognaItaly
  2. 2.Department of MathematicsUniversity of QueenslandSt. Lucia, BrisbaneAustralia

Personalised recommendations