Automation and Remote Control

, Volume 80, Issue 8, pp 1403–1418 | Cite as

Stochastic Approximation Algorithm with Randomization at the Input for Unsupervised Parameters Estimation of Gaussian Mixture Model with Sparse Parameters

  • A. A. BoiarovEmail author
  • O. N. GranichinEmail author
Stochastic Systems


We consider the possibilities of using stochastic approximation algorithms with randomization on the input under unknown but bounded interference in studying the clustering of data generated by a mixture of Gaussian distributions. The proposed algorithm, which is robust to external disturbances, allows us to process the data “on the fly” and has a high convergence rate. The operation of the algorithm is illustrated by examples of its use for clustering in various difficult conditions.


clustering unsupervised learning randomization stochastic approximation Gaussian mixture model 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



This work was partially supported by the Russian Science Foundation, project no. 16-19-00057.


  1. 1.
    Polyak, B.T., Vvedenie v optimizatsiyu, Moscow: Nauka, 1983. Translated under the title Introduction to Optimization, New York: Optimization Software, 1987.zbMATHGoogle Scholar
  2. 2.
    Robbins, H. and Monro, S., A Stochastic Approximation Method, Ann. Math. Stat., 1951, pp. 400–407.Google Scholar
  3. 3.
    Kiefer, J. and Wolfowitz, J., Stochastic Estimation of the Maximum of a Regression Function, Ann. Math. Stat., 1952, 23, 3, pp. 462–466.CrossRefzbMATHGoogle Scholar
  4. 4.
    Blum, J.R., Multidimensional Stochastic Approximation Methods, Ann. Math. Stat., 1954, 25, 4, pp. 737–744.CrossRefzbMATHGoogle Scholar
  5. 5.
    Granichin, O.N., A Stochastic Recursive Procedure with Correlated Noises in the Observation, That Employs Trial Perturbations at the Input, Vestn. Leningr. Univ., Ser. 1, 1989, no. 1(4), pp. 19–21.MathSciNetzbMATHGoogle Scholar
  6. 6.
    Granichin, O.N., Procedure of Stochastic Approximation with Disturbances at the Input, Autom. Remote Control, 1992, 53, 2, pp. 232–237.MathSciNetGoogle Scholar
  7. 7.
    Polyak, B.T. and Tsybakov, A.B., Optimal Orders of Accuracy for Search Algorithms of Stochastic Optimization, Probl. Peredachi Inf., 1990, 26, 2, pp. 45–53.Google Scholar
  8. 8.
    Rastrigin, L.A., Statisticheskie metody poiska (Statistical Search Methods), Moscow: Nauka, 1968.Google Scholar
  9. 9.
    Spall, J.C., Multivariate Stochastic Aproximation Using a Simultaneous Perturbation Gradient Aproximation, IEEE Trans. Autom. Control, 1992, vol. 37, no. 3, pp. 332–341.CrossRefGoogle Scholar
  10. 10.
    Granichin, O.N., Stochastic Approximation Search Algorithms with Randomization at the Input, Autom. Remote Control, 2015, 76, 5, pp. 762–775.MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Granichin, O., Volkovich, Z., and Toledano-Kitai, D., Randomized Algorithms in Automatic Control and Data Mining, Berlin: Springer, 2015.CrossRefGoogle Scholar
  12. 12.
    Lloyd, S., Least Squares Quantization in PCM, IEEE Trans. Inform. Theory, 1982, 28, 2, pp. 129–136.MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Shindler, M., Wong, A., and Meyerson, A., Fast and Accurate k-means For Large Datasets, Proc. 24th NIPS Conf., 2011.Google Scholar
  14. 14.
    Sculley, D., Web Scale K-Means Clustering, Proc. 19th WWW Conf., Raleigh, North Carolina, 2010.Google Scholar
  15. 15.
    Katselis, D., Beck, C.L., and van der Schaar, M., Ensemble Online Clustering through Decentralized Observations, Proc. 53rd IEEE CDC, Los Angeles, 2014, pp. 910–915.Google Scholar
  16. 16.
    Kaufman, L. and Rousseeuw, P., Finding Groups in Data: An Introduction to Cluster Analysis, New York: Wiley, 1990.CrossRefzbMATHGoogle Scholar
  17. 17.
    Granichin, O.N., and Izmakova, O.A., A Randomized Stochastic Approximation Algorithm for Self-Learning, Autom. Remote Control, 2005, 66, 8, pp. 1239–1248.MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Dempster, A., Laird, N., and Rubin, D., Maximum Likelihood from Incomplete Data via the EM Algorithm, J. Royal Stat. Soc. Ser. B, 1977, 39, 1, pp. 1–38.MathSciNetzbMATHGoogle Scholar
  19. 19.
    Bishop, C.M., Pattern Recognition and Machine Learning, New York: Springer, 2006.zbMATHGoogle Scholar
  20. 20.
    Song, M. and Wang, H., Highly Efficient Incremental Estimation of GMM for Online Data Stream Clust, Proc. SPIE III, 2005, pp. 174–184.Google Scholar
  21. 21.
    Frey, B.J. and Dueck, D., Clustering by Passing Messages between Data Points, Sci., 2007, no. 315(5814), pp. 972–976.MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Ester, M., Kriegel, H.P., Sander, J., and Xu, X., A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining, Portland, 1996, pp. 226–231.Google Scholar
  23. 23.
    Huber, P.J., Robust Statistics, New York: Wiley, 1981.CrossRefzbMATHGoogle Scholar
  24. 24.
    Hubert, L. and Arabie, P., Comparing Partitions, J. Classif., 1985, vol. 2, no. 1, pp. 193–218.CrossRefzbMATHGoogle Scholar
  25. 25.
    Dahlin, J., Wills, A., and Ninness, B., Sparse Bayesian ARX Models with Flexible Noise Distributions, Proc. 18th IFAC Sympos. on System Identification, Stockholm, 2018, pp. 25–30.Google Scholar

Copyright information

© Pleiades Publishing, Ltd. 2019

Authors and Affiliations

  1. 1.St. Petersburg State UniversitySt. PetersburgRussia
  2. 2.Institute for Problems of Mechanical EngineeringRussian Academy of SciencesSt. PetersburgRussia

Personalised recommendations