Advertisement

Alternative Variable Splitting Methods to Learn Sum-Product Networks

  • Nicola Di Mauro
  • Floriana Esposito
  • Fabrizio G. VentolaEmail author
  • Antonio Vergari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10640)

Abstract

Sum-Product Networks (SPNs) are recent deep probabilistic models providing exact and tractable inference. SPNs have been successfully employed as density estimators in several application domains. However, learning an SPN from high dimensional data still poses a challenge in terms of time complexity. This is due to the high cost of determining independencies among random variables (RVs) and sub-populations among samples, two operations that are repeated several times. Even one of the simplest greedy structure learner, LearnSPN, scales quadratically in the number of the variables to determine RVs independencies. In this work we investigate approximate but fast procedures to determine independencies among RVs whose complexity scales in sub-quadratic time. We propose two procedures: a random subspace approach and one that adopts entropy as a criterion to split RVs in linear time. Experimental results prove that LearnSPN equipped by our splitting procedures is able to reduce learning and/or inference times while preserving comparable inference accuracy.

Keywords

Machine learning Deep learning Structure learning Probabilistic models Density estimation Sum-Product Networks 

References

  1. 1.
    Adel, T., Balduzzi, D., Ghodsi, A.: Learning the structure of sum-product networks via an SVD-based algorithm. In: UAI (2015)Google Scholar
  2. 2.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  3. 3.
    Cheng, W., Kok, S., Pham, H.V., Chieu, H.L., Chai, K.M.A.: Language modeling with sum-product networks. In: INTERSPEECH 2014, pp. 2098–2102 (2014)Google Scholar
  4. 4.
    Darwiche, A.: A differential approach to inference in Bayesian networks. JACM 50(3), 280–305 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Dennis, A., Ventura, D.: Learning the architecture of sum-product networks using clustering on variables. In: NIPS 25, pp. 2033–2041 (2012)Google Scholar
  6. 6.
    Di Mauro, N., Vergari, A., Basile, T.M.A.: Learning Bayesian random cutset forests. In: Esposito, F., Pivert, O., Hacid, M.-S., Raś, Z.W., Ferilli, S. (eds.) ISMIS 2015. LNCS (LNAI), vol. 9384, pp. 122–132. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-25252-0_13 CrossRefGoogle Scholar
  7. 7.
    Di Mauro, N., Vergari, A., Esposito, F.: Learning accurate cutset networks by exploiting decomposability. In: Gavanelli, M., Lamma, E., Riguzzi, F. (eds.) AI*IA 2015. LNCS, vol. 9336, pp. 221–232. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24309-2_17 CrossRefGoogle Scholar
  8. 8.
    Friesen, A., Domingos, P.: The sum-product theorem: a foundation for learning tractable models. In: ICML, pp. 1909–1918 (2016)Google Scholar
  9. 9.
    Gens, R., Domingos, P.: Learning the structure of sum-product networks. In: ICML, pp. 873–880 (2013)Google Scholar
  10. 10.
    Haaren, J.V., Davis, J.: Markov network structure learning: a randomized feature generation approach. In: AAAI. AAAI Press (2012)Google Scholar
  11. 11.
    Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  12. 12.
    Lowd, D., Davis, J.: Learning Markov network structure with decision trees. In: ICDM, pp. 334–343. IEEE Computer Society Press (2010)Google Scholar
  13. 13.
    MacKay, D.J.C.: Information Theory, Inference & Learning Algorithms. Cambridge University Press, New York (2002)Google Scholar
  14. 14.
    Martens, J., Medabalimi, V.: On the Expressive Efficiency of Sum Product Networks. CoRR abs/1411.7717 (2014)Google Scholar
  15. 15.
    Molina, A., Natarajan, S., Kersting, K.: Poisson sum-product networks: a deep architecture for tractable multivariate poisson distributions. In: AAAI (2017)Google Scholar
  16. 16.
    Peharz, R.: Foundations of sum-product networks for probabilistic modeling. Ph.D. thesis, Graz University of Technology, SPSC (2015)Google Scholar
  17. 17.
    Peharz, R., Kapeller, G., Mowlaee, P., Pernkopf, F.: Modeling speech with sum-product networks: application to bandwidth extension. In: ICASSP (2014)Google Scholar
  18. 18.
    Poon, H., Domingos, P.: Sum-product networks: a new deep architecture. In: UAI 2011 (2011)Google Scholar
  19. 19.
    Queyranne, M.: Minimizing symmetric submodular functions. Math. Program. 82(1–2), 3–12 (1998)zbMATHMathSciNetGoogle Scholar
  20. 20.
    Rahman, T., Kothalkar, P., Gogate, V.: Cutset networks: a simple, tractable, and scalable approach for improving the accuracy of chow-liu trees. In: Calders, T., Esposito, F., Hüllermeier, E., Meo, R. (eds.) ECML PKDD 2014. LNCS, vol. 8725, pp. 630–645. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-662-44851-9_40 Google Scholar
  21. 21.
    Rooshenas, A., Lowd, D.: Learning sum-product networks with direct and indirect variable interactions. In: ICML (2014)Google Scholar
  22. 22.
    Roth, D.: On the hardness of approximate reasoning. Artif. Intell. 82(12), 273–302 (1996)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Vergari, A., Di Mauro, N., Esposito, F.: Visualizing and understanding sum-product networks. CoRR abs/1608.08266 (2016)Google Scholar
  24. 24.
    Vergari, A., Di Mauro, N., Esposito, F.: Simplifying, regularizing and strengthening sum-product network structure learning. In: Appice, A., Rodrigues, P.P., Santos Costa, V., Gama, J., Jorge, A., Soares, C. (eds.) ECML PKDD 2015. LNCS (LNAI), vol. 9285, pp. 343–358. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-23525-7_21 CrossRefGoogle Scholar
  25. 25.
    Yuan, Z., Wang, H., Wang, L., Lu, T., Palaiahnakote, S., Tan, C.L.: Modeling spatial layout for scene image understanding via a novel multiscale sum-product network. Expert Syst. Appl. 63, 231–240 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Nicola Di Mauro
    • 1
  • Floriana Esposito
    • 1
  • Fabrizio G. Ventola
    • 1
    Email author
  • Antonio Vergari
    • 1
  1. 1.Department of Computer ScienceUniversity of Bari “Aldo Moro”BariItaly

Personalised recommendations