Subgoal Reachability in Goal Conditioned Hierarchical Reinforcement Learning

Michał Bortkiewicz, Jakub Łyskawa, Paweł Wawrzyński, Paweł Wawrzyński, Mateusz Ostaszewski, Artur Grudkowski, Bartłomiej Sobieski, Tomasz Trzciński, Tomasz Trzciński, Tomasz Trzciński, Tomasz Trzciński, Tomasz Trzciński

2024

Abstract

Achieving long-term goals becomes more feasible when we break them into smaller, manageable subgoals. Yet, a crucial question arises: how specific should these subgoals be? Existing Goal-Conditioned Hierarchical Reinforcement Learning methods are based on lower-level policies aiming at subgoals designated by higher-level policies. These methods are sensitive to the proximity threshold under which the subgoals are considered achieved. Constant thresholds make the subgoals impossible to achieve in the early learning stages, easy to achieve in the late stages, and require careful manual tuning to yield reasonable overall learning performance. We argue that subgoal precision should depend on the agent’s recent performance rather than be predefined. We propose Adaptive Subgoal Required Distance (ASRD), a drop-in replacement method for subgoal threshold creation that considers the agent’s current lower-level capabilities for appropriate subgoals. Our results demonstrate that subgoal precision is essential for HRL convergence speed, and our method improves the performance of existing HRL algorithms.

Paper Citation

in Harvard Style

Bortkiewicz M., Łyskawa J., Wawrzyński P., Ostaszewski M., Grudkowski A., Sobieski B. and Trzciński T. (2024). Subgoal Reachability in Goal Conditioned Hierarchical Reinforcement Learning. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 221-230. DOI: 10.5220/0012326200003636

in Bibtex Style

@conference{icaart24,
author={Michał Bortkiewicz and Jakub Łyskawa and Paweł Wawrzyński and Mateusz Ostaszewski and Artur Grudkowski and Bartłomiej Sobieski and Tomasz Trzciński},
title={Subgoal Reachability in Goal Conditioned Hierarchical Reinforcement Learning},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART},
year={2024},
pages={221-230},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012326200003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART
TI - Subgoal Reachability in Goal Conditioned Hierarchical Reinforcement Learning
SN - 978-989-758-680-4
AU - Bortkiewicz M.
AU - Łyskawa J.
AU - Wawrzyński P.
AU - Ostaszewski M.
AU - Grudkowski A.
AU - Sobieski B.
AU - Trzciński T.
PY - 2024
SP - 221
EP - 230
DO - 10.5220/0012326200003636
PB - SciTePress