Topological Visualization Method for Understanding the Landscape of Value Functions and Structure of the State Space in Reinforcement Learning

Yuki Nakamura, Takeshi Shibuya

2020

Abstract

Reinforcement learning is a learning framework applied in various fields in which agents autonomously acquire control rules. Using this method, the designer constructs a state space and reward function and sets various parameters to obtain ideal performance. The actual performance of the agent depends on the design. Accordingly, a poor design causes poor performance. In that case, the designer needs to examine the cause of the poor performance; to do so, it is important for the designer to understand the current agent control rules. In the case where the state space is less than or equal to two dimensions, visualizing the landscape of the value function and the structure of the state space is the most powerful method to understand these rules. However, in other cases, there is no method for such a visualization. In this paper, we propose a method to visualize the landscape of the value function and the structure of the state space even when the state space has a high number of dimensions. Concretely, we employ topological data analysis for the visualization. We confirm the effectiveness of the proposed method via several numerical experiments.

Paper Citation

in Harvard Style

Nakamura Y. and Shibuya T. (2020). Topological Visualization Method for Understanding the Landscape of Value Functions and Structure of the State Space in Reinforcement Learning. In Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-395-7, pages 370-377. DOI: 10.5220/0008913303700377

in Bibtex Style

@conference{icaart20,
author={Yuki Nakamura and Takeshi Shibuya},
title={Topological Visualization Method for Understanding the Landscape of Value Functions and Structure of the State Space in Reinforcement Learning},
booktitle={Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2020},
pages={370-377},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008913303700377},
isbn={978-989-758-395-7},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Topological Visualization Method for Understanding the Landscape of Value Functions and Structure of the State Space in Reinforcement Learning
SN - 978-989-758-395-7
AU - Nakamura Y.
AU - Shibuya T.
PY - 2020
SP - 370
EP - 377
DO - 10.5220/0008913303700377