DICE: A New Family of Bivariate Estimation of Distribution Algorithms Based on Dichotomised Multivariate Gaussian Distributions

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10199)

Abstract

A new family of Estimation of Distribution Algorithms (EDAs) for discrete search spaces is presented. The proposed algorithms, which we label DICE (Discrete Correlated Estimation of distribution algorithms) are based, like previous bivariate EDAs such as MIMIC and BMDA, on bivariate marginal distribution models. However, bivariate models previously used in similar discrete EDAs were only able to exploit an O(d) subset of all the \(O(d^{2})\) bivariate variable dependencies between d variables. We introduce, and utilize in DICE, a model based on dichotomised multivariate Gaussian distributions. These models are able to capture and make use of all \(O(d^{2})\) bivariate variable interactions in binary and multary search spaces. This paper tests the performances of these new EDA models and algorithms on a suite of challenging combinatorial optimization problems, and compares their performances to previously used discrete-space bivariate EDA models. EDAs utilizing these new dichotomised Gaussian (DG) models exhibit significantly superior optimization performances, with the performance gap becoming more marked with increasing dimensionality.

Keywords

Dichotomised Gaussian models EDAs Combinatorial optimization 

Notes

Acknowledgements

This work was supported, in part, by Science Foundation Ireland grant 10/CE/I1855 to Lero - the Irish Software Engineering Research Centre (www.lero.ie).

References

  1. 1.
    Aizerman, A., Braverman, E., Rozoner, L.: Theoretical foundations of the potential function method in pattern recognition learning. Autom. Remote Control 25, 821–837 (1964)Google Scholar
  2. 2.
    Altenberg, L.: NK fitness landscapes. In: Back, T., Fogel, D., Michalewicz, Z. (eds.) Handbook of Evolutionary Computation, pp. B2.7:5–B2.7:10. Oxford University Press, New York (1997)Google Scholar
  3. 3.
    Baluja, S., Caruana, R.: Removing the genetics from the standard genetic algorithm. In: Machine Learning: Proceedings of the Twelfth International Conference, pp. 38–46 (1995)Google Scholar
  4. 4.
    Baluja, S., Davies, S.: Using optimal dependency-trees for combinational optimization. In: Proceedings of the Fourteenth International Conference on Machine Learning, pp. 30–38. Morgan Kaufmann Publishers (1997)Google Scholar
  5. 5.
    Boros, E., Hammer, P., Tavares, G.: Local search heuristics for quadratic unconstrained binary optimization (QUBO). J. Heuristics 13(2), 99–132 (2007)CrossRefGoogle Scholar
  6. 6.
    Caprara, A., Furini, F., Lodi, A., Mangia, M., Rovatti, R., Setti, G.: Generation of antipodal random vectors with prescribed non-stationary 2-nd order statistics. IEEE Trans. Sig. Process. 62(6), 1603–1612 (2014)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Chang, Y.W., Hsieh, C.J., Chang, K.W., Ringgaard, M., Lin, C.J.: Training and testing low-degree polynomial data mappings via linear SVM. J. Mach. Learn. Res. 11(Apr), 1471–1490 (2010)MathSciNetMATHGoogle Scholar
  8. 8.
    Chow, C., Liu, C.: Approximating discrete probability distributions with dependence trees. IEEE Trans. Inform. Theory 14(3), 462–467 (1968)CrossRefMATHGoogle Scholar
  9. 9.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, New York (2000)CrossRefMATHGoogle Scholar
  10. 10.
    De Bonet, J., Isbell, C., Viola, P., et al.: MIMIC: finding optima by estimating probability densities. In: Advances in Neural Information Processing Systems, pp. 424–430 (1997)Google Scholar
  11. 11.
    Emrich, L., Piedmonte, M.: A method for generating high-dimensional multivariate binary variates. Am. Stat. 45(4), 302–304 (1991)Google Scholar
  12. 12.
    Etxeberria, R., Larranaga, P.: Global optimization using Bayesian networks. In: Second Symposium on Artificial Intelligence (CIMAF-99), Habana, Cuba, pp. 332–339 (1999)Google Scholar
  13. 13.
    Gange, S.: Generating multivariate categorical variates using the iterative proportional fitting algorithm. Am. Stat. 49(2), 134–138 (1995)Google Scholar
  14. 14.
    Genz, A., Bretz, F.: Computation of Multivariate Normal and t Probabilities, vol. 195. Springer Science & Business Media, New York (2009)MATHGoogle Scholar
  15. 15.
    Glover, F., Hao, J.K., Kochenberger, G.: Polynomial unconstrained binary optimisation - part 2. Int. J. Metaheuristics 1(4), 317–354 (2011)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Goldberg, Y., Elhadad, M.: SplitSVM: fast, space-efficient, non-heuristic, polynomial kernel computation for NLP applications. In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pp. 237–240. Association for Computational Linguistics (2008)Google Scholar
  17. 17.
    González-Fernández, Y., Soto, M.: A survey of estimation of distribution algorithms based on copulas. Technical reportGoogle Scholar
  18. 18.
    Hansen, N., Kern, S.: Evaluating the CMA evolution strategy on multimodal test functions. In: International Conference on Parallel Problem Solving from Nature, pp. 282–291. Springer (2004)Google Scholar
  19. 19.
    Harik, G., Lobo, F., Goldberg, D.: The compact genetic algorithm. IEEE Trans. Evol. Comput. 3(4), 287–297 (1999)CrossRefGoogle Scholar
  20. 20.
    Harik, G., Lobo, F., Sastry, K.: Linkage learning via probabilistic modeling in the extended compact genetic algorithm(ECGA). In: Pelikan, M., Sastry, K., CantúPaz, E. (eds.) Scalable Optimization via Probabilistic Modeling, pp. 39–61. Springer, New York (2006)CrossRefGoogle Scholar
  21. 21.
    Heras, F., Larrosa, J., Oliveras, A.: MiniMaxSAT: an efficient weighted Max-SAT solver. J. Artif. Intell. Res. (JAIR) 31, 1–32 (2008)MathSciNetMATHGoogle Scholar
  22. 22.
    Higham, N.: Computing the nearest correlation matrix: a problem from finance. IMA J. Numer. Anal. 22(3), 329–343 (2002)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Hoos, H., Stützle, T.: Stochastic Local Search: Foundations & Applications. Elsevier, Amsterdam (2004)MATHGoogle Scholar
  24. 24.
    Hyrš, M., Schwarz, J.: Multivariate Gaussian copula in estimation of distribution algorithm with model migration. In: 2014 IEEE Symposium on Foundations of Computational Intelligence (FOCI), pp. 114–119. IEEE (2014)Google Scholar
  25. 25.
    Jin, R., Wang, S., Yan, F., Zhu, J.: Generating spatial correlated binary data through a copulas method. Sci. Res. 3(4), 206–212 (2015)CrossRefGoogle Scholar
  26. 26.
    Lane, F., Azad, R., Ryan, C.: Principled evolutionary algorithm design and the kernel trick. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 149–150. ACM (2016)Google Scholar
  27. 27.
    Lane, F., Azad, R., Ryan, C.: Principled evolutionary algorithm search operator design and the kernel trick. In: 2016 IEEE Symposium on Model Based Evolutionary Algorithms (IEEE MBEA 2016), part of the IEEE Symposium Series on Computational Intelligence 2016, pp. 1–9 (2016)Google Scholar
  28. 28.
    Larrañaga, P., Etxeberria, R., Lozano, J., Peña, J.: Combinatorial optimization by learning and simulation of Bayesian networks. In: Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, pp. 343–352. Morgan Kaufmann Publishers Inc. (2000)Google Scholar
  29. 29.
    Larranaga, P., Lozano, J., Bengoetxea, E.: Estimation of distribution algorithms based on multivariate normal and Gaussian networks. Technical report, EHU-KZAA-IK-1 (2001)Google Scholar
  30. 30.
    Lee, A.: Generating random binary deviates having fixed marginal distributions and specified degrees of association. Am. Stat. 47(3), 209–215 (1993)Google Scholar
  31. 31.
    Macke, J., Berens, P., Ecker, A., Tolias, A., Bethge, M.: Generating spike trains with specified correlation coefficients. Neural Comput. 21(2), 397–423 (2009)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Macke, J., Murray, I., Latham, P.: How biased are maximum entropy models? In: Advances in Neural Information Processing Systems, pp. 2034–2042 (2011)Google Scholar
  33. 33.
    Mühlenbein, H.: The equation for response to selection and its use for prediction. Evol. Comput. 5(3), 303–346 (1997)CrossRefGoogle Scholar
  34. 34.
    Pelikan, M., Goldberg, D., Cantú-Paz, E.: BOA: the Bayesian optimization algorithm. In: Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation, vol. 1, pp. 525–532. Morgan Kaufmann Publishers (1999)Google Scholar
  35. 35.
    Pelikan, M., Mühlenbein, H.: The bivariate marginal distribution algorithm. In: Roy, R., Furuhashi, T., Chawdhry, P.K. (eds.) Advances in Soft Computing, pp. 521–535. Springer, New York (1999)CrossRefGoogle Scholar
  36. 36.
    Propp, J., Wilson, D.: Exact sampling with coupled Markov chains and applications to statistical mechanics. Random Struct. Algorithms 9(1–2), 223–252 (1996)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Qi, H., Sun, D.: A quadratically convergent Newton method for computing the nearest correlation matrix. SIAM J. Matrix Anal. Appl. 28(2), 360–385 (2006)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Qi, H., Sun, D.: An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem. IMA J. Numer. Anal. 31(2), 491–511 (2011)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Rasmussen, C., Williams, C.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)MATHGoogle Scholar
  40. 40.
    Salinas-Gutiérrez, R., Hernández-Aguirre, A., Villa-Diharce, E.R.: Using copulas in estimation of distribution algorithms. In: Aguirre, A.H., Borja, R.M., Garciá, C.A.R. (eds.) MICAI 2009. LNCS (LNAI), vol. 5845, pp. 658–668. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-05258-3_58CrossRefGoogle Scholar
  41. 41.
    Zhang, Q., Sun, J., Tsang, E., Ford, J.: Estimation of distribution algorithm based on mixture. Technical reportGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Fergal Lane
    • 1
  • R. Muhammad Atif Azad
    • 2
  • Conor Ryan
    • 1
  1. 1.CSIS DepartmentUniversity of LimerickLimerickIreland
  2. 2.School of Computing and Digital TechnologyBirmingham City UniversityBirminghamUK

Personalised recommendations