Nested Rollout Policy Adaptation for Multiagent System Optimization in Manufacturing

Stefan Edelkamp, Christoph Greulich


In manufacturing there are not only flow lines with stations arranged one behind the other, but also more complex networks of stations where assembly operations are performed. The considerable difference from sequential flow lines is that a partially ordered set of required components are brought together in order to form a single unit at the assembly stations in a competitive multiagent system scenario. In this paper we optimize multiagent control for such flow production units with recent advances of Nested Monte-Carlo Search. The optimization problem is implemented as a single-agent game in a generic search framework. In particular, we employ Nested Monte-Carlo Search with Rollout Policy Adaptation and apply it to a modern flow production unit, comparing it to solutions obtained with a simulator and with a model checker.


  1. Bhat, U. (1986). Finite capacity assembly-like queues. Queueing Systems, 1:85-101.
  2. Bouzy, B. (2016). An experimental investigation on the pancake problem. In Computer Games: Fourth Workshop on Computer Games, pages 30-43, Cham. Springer International Publishing.
  3. Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. (2004). A survey of Monte Carlo tree search methods. 4(1):1-43.
  4. Bürckert, H.-J., Fischer, K., and Vierke, G. (2000). Holonic transport scheduling with teletruck. Applied Artificial Intelligence, 14(7):697-725.
  5. Burman, M. (1995). New results in flow line analysis . PhD thesis, MIT.
  6. Cazenave, T. (2009). Nested monte-carlo search. In IJCAI, pages 456-461.
  7. Cazenave, T. and Teytaud, F. (2012a). Application of the Nested Rollout Policy Adaptation Algorithm to the Traveling Salesman Problem with Time Windows, pages 42-54. Springer.
  8. Cazenave, T. and Teytaud, F. (2012b). Beam nested rollout policy adaptation. In ECAI-Workshop on Computer Games, pages 1-12.
  9. Dorer, K. and Calisti, M. (2005). An adaptive solution to dynamic transport optimization. In Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems, pages 45-51. ACM.
  10. Edelkamp, S. and Cazenave, T. (2016). Improved diversity in nested rollout policy adaptation. In German Conference on AI (KI 2016).
  11. Edelkamp, S. and Greulich, C. (2016). Using SPIN for the optimized scheduling of discrete event systems in manufacturing. In SPIN 2016, pages 57-77. Springer.
  12. Fischer, K., Müller, J. R. P., and Pischel, M. (1996). Cooperative transportation scheduling: an application domain for dai. Applied Artificial Intelligence, 10(1):1- 34.
  13. Ganji, F., Morales Kluge, E., and Scholz-Reiter, B. (2010). Bringing Agents into Application: Intelligent Products in Autonomous Logistics. In Artificial intelligence and Logistics (AiLog) - Workshop at ECAI 2010, pages 37-42.
  14. Gomes, C. P., Selman, B., Crato, N., and Kautz, H. (2000). Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Autom. Reason., 24(1-2):67-100.
  15. Greulich, C. and Edelkamp, S. (2016). Branch-and-bound optimization of a multiagent system for flow production using model checking. In ICAART 2016.
  16. Greulich, C., Edelkamp, S., and Eicke, N. (2015). Cyberphysical multiagent simulation in production logistics. In MATES 2015.
  17. Harrison, J. (1973). Assembly-like queues. Journal of Applied Probability, 10:354-367.
  18. Himoff, J., Rzevski, G., and Skobelev, P. (2006). Magenta technology multi-agent logistics i-scheduler for road transportation. In AAMAS 06, pages 1514-1521. ACM.
  19. Hopp, W. and Simon, J. (1989). Bounds and heuristics for assembly-like queues. Queueing Systems, 4:137-156.
  20. Huang, S.-C., Arneson, B., Hayward, R. B., Mueller, M., and Pawlewicz, J. (2013). Mohex 2.0: A pattern-based MCTS Hex player. In Computers and Games, pages 60-71.
  21. Kocsis, L. and Szepesvári, C. (2006). Bandit based MonteCarlo planning. In ECML, pages 282-293.
  22. Lipper, E. and Sengupta, E. (1986). Assembly-like queues with finite capacity: bounds, asymptotics and approximations. Queueing Systems, pages 67-83.
  23. Manitz, M. (2008). Queueing-model based analysis of assembly lines with finite buffers and general service times. Computers & Operations Research, 35(8):2520 - 2536.
  24. Morales Kluge, E., Ganji, F., and Scholz-Reiter, B. (2010). Intelligent products - towards autonomous logistic processes - a work in progress paper. In Intern. PLM Conf.
  25. Parragh, S. N., Doerner, K. F., and Hartl, R. F. (2008). A Survey on Pickup and Delivery Problems Part II: Transportation between Pickup and Delivery Locations. Journal für Betriebswirtschaft, 58(2):81-117.
  26. Rosin, C. D. (2011). Nested rollout policy adaptation for monte carlo tree search. In IJCAI, pages 649-654.
  27. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503.

Paper Citation

in Harvard Style

Edelkamp S. and Greulich C. (2017). Nested Rollout Policy Adaptation for Multiagent System Optimization in Manufacturing . In Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART, ISBN 978-989-758-219-6, pages 284-290. DOI: 10.5220/0006204502840290

in Bibtex Style

author={Stefan Edelkamp and Christoph Greulich},
title={Nested Rollout Policy Adaptation for Multiagent System Optimization in Manufacturing},
booktitle={Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,},

in EndNote Style

JO - Proceedings of the 9th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,
TI - Nested Rollout Policy Adaptation for Multiagent System Optimization in Manufacturing
SN - 978-989-758-219-6
AU - Edelkamp S.
AU - Greulich C.
PY - 2017
SP - 284
EP - 290
DO - 10.5220/0006204502840290