ADAPTIVE STATE REPRESENTATIONS FOR MULTI-AGENT REINFORCEMENT LEARNING

Yann-Michaël De Hauwere, Peter Vrancx, Ann Nowé

2011

Abstract

When multiple agents act in the same environment, single-agent reinforcement learning (RL) techniques often fail, as they do not take into account other agents. An agent using single agent RL generally does not have sufficient information to obtain a good policy. However, multi-agent techniques that simply extend the state space to include information on the other agents suffer from a large overhead, leading to very slow learning. In this paper we describe a multi-level RL algorithm which acts independently whenever possible and learns in which states it should enrich its state information with information about other agents. Such states, which we call conflict states are detected using statistical information about expected payoffs in these states. We demonstrate through experiments that our approach learns a good trade-off between learning in the single-agent state space and learning in the multi-agent state space.

References

  1. Boutilier, C. (1996). Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th Conference on Theoretical Aspects of Rationality and Knowledge, pages 195-210, Renesse, Holland.
  2. Claus, C. and Boutilier, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 746-752. AAAI Press.
  3. De Hauwere, Y., Vrancx, P., and Nowé, A. (2009). Multilayer learning and knowledge transfer in mas. In Proceedings of the 7th European Workshop on MultiAgent Systems.
  4. De Hauwere, Y., Vrancx, P., and Nowé, A. (2010). Learning multi-agent state space representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multi-Agent Systems.
  5. Greenwald, A. and Hall, K. (2003). Correlated-q learning. In AAAI Spring Symposium, pages 242-249. AAAI Press.
  6. Hu, J. and Wellman, M. (2003). Nash q-learning for general-sum stochastic games. Journal of Machine Learning Research, 4:1039-1069.
  7. Kok, J., 't Hoen, P., Bakker, B., and Vlassis, N. (2005). Utile coordination: Learning interdependencies among cooperative agents. In Proceedings of the IEEE Symposium on Computational Intelligence and Games (CIG05), pages 29-36.
  8. Kok, J. and Vlassis, N. (2004). Sparse cooperative qlearning. In Proceedings of the 21st international conference on Machine learning. ACM New York, NY, USA.
  9. Melo, F. and Veloso, M. (2009). Learning of coordination: Exploiting sparse interactions in multiagent systems. In Proceedings of the 8th International Conference on Autonomous Agents and Multi-Agent Systems.
  10. Spaan, M. and Melo, F. (2008). Interaction-driven markov games for decentralized multiagent planning under uncertainty. In Padgham, Parkes, Müller, and Parsons, editors, Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems, pages 525-532.
  11. Sutton, R. and Barto, A. (1998). Reinforcement Learning: An Introduction. MIT Press.
  12. Tsitsiklis, J. (1994). Asynchronous stochastic approximation and q-learning. Journal of Machine Learning, 16(3):185-202.
  13. Vrancx, P. (2010). Decentralised Reinforcement Learning in Markov games. PhD thesis, Vrije Universiteit Brussel.
  14. Watkins, C. (1989). Learning from Delayed Rewards. PhD thesis, University of Cambridge.
Download


Paper Citation


in Harvard Style

De Hauwere Y., Vrancx P. and Nowé A. (2011). ADAPTIVE STATE REPRESENTATIONS FOR MULTI-AGENT REINFORCEMENT LEARNING . In Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-8425-41-6, pages 181-189. DOI: 10.5220/0003145701810189


in Bibtex Style

@conference{icaart11,
author={Yann-Michaël De Hauwere and Peter Vrancx and Ann Nowé},
title={ADAPTIVE STATE REPRESENTATIONS FOR MULTI-AGENT REINFORCEMENT LEARNING},
booktitle={Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2011},
pages={181-189},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003145701810189},
isbn={978-989-8425-41-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - ADAPTIVE STATE REPRESENTATIONS FOR MULTI-AGENT REINFORCEMENT LEARNING
SN - 978-989-8425-41-6
AU - De Hauwere Y.
AU - Vrancx P.
AU - Nowé A.
PY - 2011
SP - 181
EP - 189
DO - 10.5220/0003145701810189