A Reinforcement Learning Based Model for Adaptive Service Quality Management in E-Commerce Websites

  • Hoda Ghavamipoor
  • S. Alireza Hashemi GolpayeganiEmail author
Research Paper


Providing high-quality service to all users is a difficult and inefficient strategy for e-commerce providers, especially when Web servers experience overload conditions that cause increased response time and request rejections, leading to user frustration and reduced revenue. In an e-commerce system, customer Web sessions have differing values for service providers. These tend to: give preference to customer Web sessions that are likely to bring more profit by providing better service quality. This paper proposes a reinforcement-learning based adaptive e-commerce system model that adapts the service quality level for different Web sessions within the customer’s navigation in order to maximize total profit. The e-commerce system is considered as an electronic supply chain which includes a network of basic e- providers used to supply e-commerce services for end customers. The learner agent noted as e-commerce supply chain manager (ECSCM) agent allocates a service quality level to the customer’s request based on his/her navigation pattern in the e-commerce Website and selects an optimized combination of service providers to respond to the customer’s request. To evaluate the proposed model, a multi agent framework composed of three agent types, the ECSCM agent, customer agent (buyer/browser) and service provider agent, is employed. Experimental results show that the proposed model improves total profits through cost reduction and revenue enhancement simultaneously and encourages customers to purchase from the Website through service quality adaptation.


Electronic commerce supply chain Quality of service Adaptive system Multi agent systems Reinforcement learning 


  1. Aciar S, Zhang D, Simoff S, Debenham J (2007) Informed recommender: basing recommendations on consumer product reviews. IEEE Intell Syst 22(3):39–47CrossRefGoogle Scholar
  2. Al-Masri E, Mahmoud QH (2007) QoS-based discovery and ranking of web services. In: 16th International conference on computer communications and networks. IEEE, pp 529–534Google Scholar
  3. Bathumalai G (2008) Self-adapting websites: mining user access logs. Dissertation, Robert Gordon University, AberdeenGoogle Scholar
  4. Belk M, Fidas C, Germanakos P, Samaras G (2015) Do human cognitive differences in information processing affect preference and performance of CAPTCHA? Int J Hum Comput Stud 84:1–18CrossRefGoogle Scholar
  5. Bellifemine F, Poggi A, Rimassa G (2001) JADE: a FIPA2000 compliant agent development environment. In: 5th International conference on autonomous agents. ACM, pp 216–217Google Scholar
  6. Bhatti N, Friedrich R (1999) Web server support for tiered services. IEEE Netw 13(5):64–71CrossRefGoogle Scholar
  7. Brusilovsky P, Kobsa A, Nejdl W (2007) The adaptive web. Lecture notes in computer science, vol 4321. Springer, Cham, pp 325–341Google Scholar
  8. Chan NN, Gaaloul W, Tata S (2012) A recommender system based on historical usage data for web service discovery. Serv Oriented Comput Appl 6(1):51–63CrossRefGoogle Scholar
  9. Chen M, Ryu YU (2013) Facilitating effective user navigation through website structure improvement. IEEE Trans Knowl Data Eng 25(3):571–588CrossRefGoogle Scholar
  10. Chen CC, Huang TC, Park JJ, Yen NY (2015) Real-time smartphone sensing and recommendations towards context-awareness shopping. Multimed Syst 21(1):61–72CrossRefGoogle Scholar
  11. Ewing JM, Menascé DA (2009) Business-oriented autonomic load balancing for multi-tiered Web sites. In: IEEE international symposium on modeling, analysis and simulation of computer and telecommunication systems. IEEE, pp 1–10Google Scholar
  12. Ghavamipoor H, Golpayegani SAH (2013) Comparing and applying the approach of supply chain in electronic services management. Int J Comput Inf Technol 1(2):118–136Google Scholar
  13. Ghavamipoor H, Golpayegani SAH (2016) QoS-aware provider selection in e-services supply chain. In: 8th International conference on information and knowledge technology. IEEE, pp 258–262Google Scholar
  14. Ghavamipoor H, Golpayegani SAH (2017) A QoS sensitive model for e-commerce customer behavior. J Res Interact Mark 11(4):380–397CrossRefGoogle Scholar
  15. Harini N, Padmanabhan TR (2013) Admission control and request scheduling for secured-concurrent-available architecture. Int J Comput Appl 63(6):24–30Google Scholar
  16. Hong J, Suh EH, Kim J, Kim S (2009) Context-aware system for proactive personalized service based on context history. Expert Syst Appl 36(4):7448–7457CrossRefGoogle Scholar
  17. Lakshmi MS, Kumar SP, Janardhan M, Gayathri K (2017) Machine learning methods for refining SLA based admission control and resource allocation in cloud computing. Int J Adv Res Comput Sci 8(9):834–840CrossRefGoogle Scholar
  18. Larisa G, Mariia S, Andriy R (2014) Control strategy of the input stream on the online charging system in peak load moments. In: 24th International crimean conference microwave and telecommunication technology. IEEE, pp 312–313Google Scholar
  19. Lee Y, Kozar KA (2006) Investigating the effect of website quality on e-business success: an analytic hierarchy process (AHP) approach. Decis Support Syst 42(3):1383–1401CrossRefGoogle Scholar
  20. Li K, Jamin S (2000) A measurement-based admission-controlled web server. In: Proceedings of the 19th annual joint conference of the IEEE computer and communications societies, vol 2, pp 651–659Google Scholar
  21. Li YM, Wu CT, Lai CY (2013) A social recommender mechanism for e-commerce: combining similarity, trust, and relationship. Decis Support Syst 55(3):740–752CrossRefGoogle Scholar
  22. Lin HF (2007) The impact of Website quality dimensions on customer satisfaction in the B2C e-commerce context. Total Qual Manag Bus Excell 18(4):363–378CrossRefGoogle Scholar
  23. Mark K, Csaba L (2007) Analyzing customer behavior model graph (CBMG) using Markov chains. In: 11th International conference on intelligent engineering systems. IEEE, pp 71–76Google Scholar
  24. Menascé DA (2002) QoS issues in web services. IEEE Internet Comput 6(6):72–75CrossRefGoogle Scholar
  25. Poggi N (2014) AUGURES: profit-aware web infrastructure management. Doctoral thesis, Polytechnic University of CataloniaGoogle Scholar
  26. Poggi N, Carrera D, Gavalda R, Ayguadé E (2011) Non-intrusive estimation of QoS degradation impact on e-commerce user satisfaction. In: 10th IEEE international symposium on network computing and applications. IEEE, pp 179–186Google Scholar
  27. Raufi B, Georgieva J, Luma A, Ismaili F, Zenuni X (2010) An adaptation algorithm for adaptive Web based systems based on link structure and document similarity. In: 9th WSEAS international conference on telecommunications and informatics. World Scientific and Engineering Academy and Society, pp 29–34Google Scholar
  28. Rosaci D, Sarné GM (2012) A multi-agent recommender system for supporting device adaptivity in e-commerce. J Intell Inf Syst 38(2):393–418CrossRefGoogle Scholar
  29. Sarwar B et al (2001) Item-based collaborative filtering recommendation algorithms. In: 10th international conference on world wide web. ACM, pp 285–295Google Scholar
  30. Schafer JB et al (2007) Collaborative filtering recommender systems. The adaptive web. Springer, Heidelberg, pp 291–324Google Scholar
  31. Suchacka G, Borzemski L (2013) Web server support for e-customer loyalty through QoS differentiation. In: Nguyen NT (eds) Transactions on computational collective intelligence XII. Lecture Notes in Computer Science. vol. 8240. Springer, HeidelbergGoogle Scholar
  32. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  33. Totok A, Karamcheti V (2010) RDRP: reward-driven request prioritization for e-commerce web sites. Electron Commer Res Appl 9(6):549–561CrossRefGoogle Scholar
  34. Urgaonkar B (2005) Dynamic resource management in internet hosting platforms. Doctoral dissertation, University of Massachusetts AmherstGoogle Scholar
  35. Urgaonkar B, Shenoy P (2005) Cataclysm: policing extreme overloads in internet applications. In: 14th International conference on world wide web. ACM, pp 740–749Google Scholar
  36. Weiss G (1999) Multiagent systems: a modern approach to distributed artificial intelligence. MIT Press, CambridgeGoogle Scholar
  37. Yin PY, Guo YM (2013) Optimization of multi-criteria website structure based on enhanced tabu search and web usage mining. Appl Math Comput 219(24):11082–11095Google Scholar
  38. Yue C, Wang H (2007) Profit-aware admission control for overload protection in e-commerce web sites. In: 15th IEEE international workshop on quality of service. IEEE, pp 188–193Google Scholar
  39. Zatwarnicki K, Zatwarnicka A (2014) The cluster-based time-aware web system. In: International conference on computer networks. Springer, pp 37–46Google Scholar
  40. Zheng Z, Zhang Y, Lyu MR (2014) Investigating QoS of real-world web services. IEEE Trans Serv Comput 7(1):32–39CrossRefGoogle Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  • Hoda Ghavamipoor
    • 1
  • S. Alireza Hashemi Golpayegani
    • 1
    Email author
  1. 1.Computer and Information Technology Engineering DepartmentAmirkabir University (Tehran Polytechnic)TehranIran

Personalised recommendations