Abstract
Many techniques that boost the speed or quality of metaheuristic search have been reported within literature. The present contribution investigates the rather rare combination of reinforcement learning and metaheuristics. Reinforcement learning techniques describe how an autonomous agent can learn from experience. Previous work has shown that a network of simple reinforcement learning devices based on learning automata can generate good heuristics for (multi) project scheduling problems. However, using reinforcement learning to generate heuristics is just one method of how reinforcement learning can strengthen metaheuristic search. Both existing and new methodologies to boost metaheuristics using reinforcement learning are presented together with experiments on actual benchmarks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bai, R., Burke, E.K., Gendreau, M., Kendall, G., Mccollum, B.: Memory length in hyper-heuristics: An empirical study. In: Proceedings of the 2007 IEEE Symposium on Computational Intelligence in Scheduling, CI-Sched 2007 (2007)
Battiti, R., Brunato, M., Mascia, F.: Reactive Search and Intelligent Optimization. Operations research/Computer Science Interfaces, vol. 45. Springer (2008)
Bennett, K.P., Parrado-Hernández, E.: The interplay of optimization and machine learning research. J. Mach. Learn. Res. 7, 1265–1281 (2006)
Boyan, J.: Learning Evaluation Functions for Global Optimization. PhD thesis, Carnegie-Mellon University (1998)
Boyan, J., Moore, A.W., Kaelbling, P.: Learning evaluation functions to improve optimization by local search. Journal of Machine Learning Research 1, 2000 (2000)
Burke, E., Hart, E., Kendall, G., Newall, J., Ross, P., Schulenburg, S.: Hyper-heuristics: An emerging direction in modern search technology. In: Handbook of Metaheuristics, pp. 457–474. Kluwer Academic Publishers (2003)
Burke, E.K., Kendall, G., Soubeiga, E.: A tabu-search hyperheuristic for timetabling and rostering. Journal of Heuristics 9, 451–470 (2003)
Ceschia, S., Schaerf, A.: Local search and lower bounds for the patient admission scheduling problem. Computers & Operartions Research 38, 1452–1463 (2011)
Confessore, G., Giordani, S., Rismondo, S.: A market-based multi-agent system model for decentralized multi-project scheduling. Annals of Operational Research 150, 115–135 (2007)
Demaine, E.D., Demaine, M.L.: Jigsaw puzzles, edge matching, and polyomino packing: Connections and complexity. Graphs and Combinatorics 23, 195–208 (2007); Special issue on Computational Geometry and Graph Theory: The Akiyama-Chvatal Festschrift
Demeester. P.: Patient admission scheduling website (2009), http://allserv.kahosl.be/~peter/pas/ (last visit August 15, 2011)
Demeester, P., De Causmaecker, P., Vanden Berghe, G.: Applying a local search algorithm to automatically assign patients to beds. In: Proceedings of the 22nd Conference on Quantitive Decision Making (Orbel 22), pp. 35–36 (2008)
Demeester, P., Souffriau, W., De Causmaecker, P., Vanden Berghe, G.: A hybrid tabu search algorithm for automatically assigning patients to beds. Artif. Intell. Med. 48, 61–70 (2010)
Dietterich, T.G., Zhang, W.: Solving combinatorial optimization tasks by reinforcement learning: A general methodology applied to resource-constrained scheduling. Journal of Artificial Intelligence Research (2000)
Gabel, T.: Multi-agent Reinforcement Learning Approaches for Distributed Job-Shop Scheduling Problems. PhD thesis, Universität Osnabrück, Deutschland (2009)
Gambardella, L.M., Dorigo, M.: Ant-q: A réinforcement learning approach to the traveling salesman problem, pp. 252–260. Morgan Kaufmann (1995)
Glover, F., Kochenberger, G.A.: Handbook of metaheuristics. Springer (2003)
Homberger, J.: A (μ, λ)-coordination mechanism for agent-based multi-project scheduling. OR Spectrum (2009), doi:10.1007/s00291-009-0178-3
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Proceedings of the Eleventh International Conference on Machine Learning, pp. 157–163. Morgan Kaufmann (1994)
Miagkikh, V.V., Punch III, W.F.: An approach to solving combinatorial optimization problems using a population of reinforcement learning agents (1999)
Misir, M., Wauters, T., Verbeeck, K., Vanden Berghe, G.: A new learning hyper-heuristic for the traveling tournament problem. In: Proceedings of Metaheuristic International Conference (2009)
Moll, R., Barto, A.G., Perkins, T.J., Sutton, R.S.: Learning instance-independent value functions to enhance local search. In: Advances in Neural Information Processing Systems, pp. 1017–1023. MIT Press (1998)
Narendra, K., Thathachar, M.: Learning Automata: An Introduction. Prentice-Hall International, Inc. (1989)
Nareyek, A.: Choosing search heuristics by non-stationary reinforcement learning. In: Metaheuristics: Computer Decision-Making, pp. 523–544. Kluwer Academic Publishers (2001)
Özcan, E., Misir, M., Ochoa, G., Burke, E.K.: A reinforcement learning - great-deluge hyper-heuristic for examination timetabling. Int. J. of Applied Metaheuristic Computing, 39–59 (2010)
Rummery, G.A., Niranjan, M.: On-line q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Engineering Department, Cambridge University (1994)
Richard, S., Sutton, R.S.: Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems: Proceedings of the 1995 Conference, pp. 1038–1044. MIT Press, Cambridge (1996)
Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction. MIT Press (1998)
Talbi, E.-G.: Metaheuristics: From Design to Implementation. John Wiley and Sons (2009)
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)
Taylor, M.E., Stone, P., Liu, Y.: Transfer learning via inter-task mappings for temporal difference learning. Journal of Machine Learning Research 8(1), 2125–2167 (2007)
Thathachar, M.A.L., Sastry, P.S.: Networks of Learning Automata: Techniques for Online Stochastic Optimization. Kluwer Academic Publishers (2004)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Machine Learning 8, 279–292 (1992)
Watkins, C.J.C.H.: Learning from Delayed Rewards. PhD thesis, Cambridge University (1989)
Wauters, T., Verbeeck, K., De Causmaecker, P., Vanden Berghe, G.: A game theoretic approach to decentralized multi-project scheduling (extended abstract). In: Proc. of 9th Int. Conf. on Autonomous Agents and Multiagent Systems, AAMAS 2010, vol. R24 (2010)
Wauters, T., Verbeeck, K., Vanden Berghe, G., De Causmaecker, P.: Learning agents for the multi-mode project scheduling problem. Journal of the Operational Research Society 62(2), 281–290 (2011)
Wauters, T., Verstichel, J., Verbeeck, K., Vanden Berghe, G.: A learning metaheuristic for the multi mode resource constrained project scheduling problem. In: Proceedings of the Third Learning and Intelligent OptimizatioN Conference, LION3 (2009)
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 229–256 (1992)
Zhang, W., Dietterich, T.: A reinforcement learning approach to job-shop scheduling. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1114–1120. Morgan Kaufmann (1995)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Wauters, T., Verbeeck, K., De Causmaecker, P., Vanden Berghe, G. (2013). Boosting Metaheuristic Search Using Reinforcement Learning. In: Talbi, EG. (eds) Hybrid Metaheuristics. Studies in Computational Intelligence, vol 434. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30671-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-642-30671-6_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-30670-9
Online ISBN: 978-3-642-30671-6
eBook Packages: EngineeringEngineering (R0)