Abstract
We present ContainerGym, a benchmark for reinforcement learning inspired by a real-world industrial resource allocation task. The proposed benchmark encodes a range of challenges commonly encountered in real-world sequential decision making problems, such as uncertainty. It can be configured to instantiate problems of varying degrees of difficulty, e.g., in terms of variable dimensionality. Our benchmark differs from other reinforcement learning benchmarks, including the ones aiming to encode real-world difficulties, in that it is directly derived from a real-world industrial problem, which underwent minimal simplification and streamlining. It is sufficiently versatile to evaluate reinforcement learning algorithms on any real-world problem that fits our resource allocation framework. We provide results of standard baseline methods. Going beyond the usual training reward curves, our results and the statistical tools used to interpret them allow to highlight interesting limitations of well-known deep reinforcement learning algorithms, namely PPO, TRPO and DQN.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We use the terms “episode” and “rollout” interchangeably in this paper.
- 2.
The ContainerGym software is available on the following GitHub repository: https://github.com/Pendu/ContainerGym.
- 3.
Increasing the timestep length \(\delta \) should be done carefully. Otherwise, the problem could become trivial. In our case, we choose \(\delta \) such that it is smaller than the minimum time it takes a PU to process the volume equivalent to one product.
References
Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)
Brockman, G., et al.: Openai gym (2016)
Compare, M., Bellani, L., Cobelli, E., Zio, E.: Reinforcement learning-based flow management of gas turbine parts under stochastic failures. Int. J. Adv. Manuf. Technol. 99(9–12), 2981–2992 (2018)
Degrave, J., et al.: Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602(7897), 414–419 (2022)
Dulac-Arnold, G., et al.: An empirical investigation of the challenges of real-world reinforcement learning. CoRR arxiv:2003.11881 (2020)
Haarnoja, T., et al.: Soft actor-critic algorithms and applications. CoRR arxiv:1812.05905 (2018)
Hein, D., et al.: A benchmark environment motivated by industrial control problems. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE (2017)
Lazic, N., et al.: Data center cooling using model-predictive control. In: Proceedings of the Thirty-Second Conference on Neural Information Processing Systems (NeurIPS-2018), Montreal, QC, pp. 3818–3827 (2018)
Mnih, V., et al.: Playing atari with deep reinforcement learning. CoRR arxiv:1312.5602 (2013)
Osiński, B., et al.: Simulation-based reinforcement learning for real-world autonomous driving. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 6411–6418 (2020)
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22(268), 1–8 (2021)
Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust region policy optimization. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 1889–1897. PMLR (2015)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. CoRR arxiv:1707.06347 (2017)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)
Todorov, E., Erez, T., Tassa, Y.: Mujoco: a physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033 (2012)
Vinyals, O., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575(7782), 350–354 (2019)
Acknowledgements
This work was funded by the German federal ministry of economic affairs and climate action through the “ecoKI” grant.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pendyala, A., Dettmer, J., Glasmachers, T., Atamna, A. (2024). ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource Allocation. In: Nicosia, G., Ojha, V., La Malfa, E., La Malfa, G., Pardalos, P.M., Umeton, R. (eds) Machine Learning, Optimization, and Data Science. LOD 2023. Lecture Notes in Computer Science, vol 14505. Springer, Cham. https://doi.org/10.1007/978-3-031-53969-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-53969-5_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53968-8
Online ISBN: 978-3-031-53969-5
eBook Packages: Computer ScienceComputer Science (R0)