Multi-Objective Service Composition Using Reinforcement Learning

  • Ahmed Moustafa
  • Minjie Zhang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8274)


Web services have the potential to offer the enterprises with the ability to compose internal and external business services in order to accomplish complex processes. Service composition then becomes an increasingly challenging issue when complex and critical applications are built upon services with different QoS criteria. However, most of the existing QoS-aware compositions are simply based on the assumption that multiple criteria, no matter whether these multiple criteria are conflicting or not, can be combined into a single criterion to be optimized, according to some utility functions. In practice, this can be very difficult as utility functions or weights are not well known a priori. In this paper, a novel multi-objective approach is proposed to handle QoS-aware Web service composition with conflicting objectives and various restrictions on quality matrices. The proposed approach uses reinforcement learning to deal with the uncertainty characteristic inherent in open and decentralized environments. Experimental results reveal the ability of the proposed approach to find a set of Pareto optimal solutions, which have the equivalent quality to satisfy multiple QoS-objectives with different user preferences.


Web services multi-objective optimization reinforcement learning 


  1. 1.
    Berbner, R., Spahn, M., Repp, N., Heckmann, O., Steinmetz, R.: Heuristics for qos-aware web service composition. In: International Conference on Web Services, ICWS 2006, pp. 72–82 (2006)Google Scholar
  2. 2.
    Canfora, G., Di Penta, M., Esposito, R., Villani, M.L.: An approach for qos-aware service composition based on genetic algorithms. In: Proceedings of the 2005 Conference on Genetic and Evolutionary Computation GECCO 2005, pp. 1069–1075. ACM, New York (2005)CrossRefGoogle Scholar
  3. 3.
    Cao, J., Sun, X., Zheng, X., Liu, B., Mao, B.: Efficient multi-objective services selection algorithm based on particle swarm optimization. In: 2010 IEEE Asia-Pacific Services Computing Conference (APSCC), pp. 603–608 (2010)Google Scholar
  4. 4.
    Chiu, D., Agrawal, G.: Cost and accuracy aware scientific workflow composition for service-oriented environments. IEEE Trans. Services Computing (2012)Google Scholar
  5. 5.
    Claro, D.B., Albers, P., Hao, J.K.: Selecting web services for optimal composition. In: SDWP 2005, pp. 32–45 (2005)Google Scholar
  6. 6.
    de Campos, A., Pozo, A.T.R., Vergilio, S.R., Savegnago, T.: Many-objective evolutionary algorithms in the composition of web services. In: 2010 Eleventh Brazilian Symposium on Neural Networks (SBRN), pp. 152–157 (2010)Google Scholar
  7. 7.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation 6(2), 182–197 (2002)CrossRefGoogle Scholar
  8. 8.
    Dehousse, S., Faulkner, S., Herssens, C., Jureta, I.J., Saerens, M.: Learning optimal web service selections in dynamic environments when many quality-of-service criteria matter. Machine Learning, InTech., 207–229 (2009)Google Scholar
  9. 9.
    Kalasapur, S., Kumar, M., Shirazi, B.A.: Dynamic service composition in pervasive computing. IEEE Trans. Parallel and Distributed Systems 18(7), 907–918 (2007)CrossRefGoogle Scholar
  10. 10.
    Lin, W., Dou, W., Luo, X., Chen, J.: A history record-based service optimization method for qos-aware service composition. In: 2011 IEEE International Conference on Web Services (ICWS), pp. 666–673 (2011)Google Scholar
  11. 11.
    Liu, C., Xu, X., Hu, D.: Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews PP(99), 1–13 (2013)Google Scholar
  12. 12.
    Mastronarde, N., Kanoun, K., Atienza, D., Frossard, P., van der Schaar, M.: Markov decision process based energy-efficient on-line scheduling for slice-parallel video decoders on multicore systems. IEEE Trans. Multimedia 15(2), 268–278 (2013)CrossRefGoogle Scholar
  13. 13.
    Maximilien, E.M., Singh, M.P.: A framework and ontology for dynamic web services selection. IEEE Internet Computing 8(5), 84–93 (2004)CrossRefGoogle Scholar
  14. 14.
    Suciu, M., Pallez, D., Cremene, M., Dumitrescu, D.: Adaptive moea/d for qos-based web service composition. In: Middendorf, M., Blum, C. (eds.) EvoCOP 2013. LNCS, vol. 7832, pp. 73–84. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  15. 15.
    Sutton, R.S., Barto, A.G.: Reinforcement learning: Introduction (1998)Google Scholar
  16. 16.
    Taboada, H.A., Espiritu, J.F., Coit, D.W.: Moms-ga: A multi-objective multi-state genetic algorithm for system reliability optimization design problems. IEEE Transactions on Reliability 57(1), 182–191 (2008)CrossRefGoogle Scholar
  17. 17.
    Wada, H., Suzuki, J., Yamano, Y., Oba, K.: E3: A multiobjective optimization framework for sla-aware service composition. IEEE Transactions on Services Computing 5(3), 358–372 (2012)CrossRefGoogle Scholar
  18. 18.
    Wang, H., Zhou, X., Zhou, X., Liu, W., Li, W., Bouguettaya, A.: Adaptive service composition based on reinforcement learning. In: Maglio, P.P., Weske, M., Yang, J., Fantinato, M. (eds.) ICSOC 2010. LNCS, vol. 6470, pp. 92–107. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  19. 19.
    Wang, S., Zheng, Z., Sun, Q., Zou, H., Yang, F.: Cloud model for service selection. In: 2011 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 666–671 (2011)Google Scholar
  20. 20.
    Watkins, C.: Learning from Delayed Rewards. PhD thesis, Cambridge University, England (1989)Google Scholar
  21. 21.
    Yu, T., Lin, K.-J.: Service selection algorithms for composing complex services with multiple qos constraints. In: Benatallah, B., Casati, F., Traverso, P. (eds.) ICSOC 2005. LNCS, vol. 3826, pp. 130–143. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  22. 22.
    Zeng, L., Benatallah, B., Ngu, A.H.H., Dumas, M., Kalagnanam, J., Chang, H.: Qos-aware middleware for web services composition. IEEE Transactions on Software Engineering 30(5), 311–327 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ahmed Moustafa
    • 1
  • Minjie Zhang
    • 1
  1. 1.School of Computer Science and Software EngineeringUniversity of WollongongGwnnevilleAustralia

Personalised recommendations