Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback

Rustam Zayanov, Francisco Melo, Manuel Lopes

2024

Abstract

We study the problem of teaching via demonstrations in sequential decision-making tasks. In particular, we focus on the situation when the teacher has no access to the learner’s model and policy, and the feedback from the learner is limited to trajectories that start from states selected by the teacher. The necessity to select the starting states and infer the learner’s policy creates an opportunity for using the methods of inverse reinforcement learning and active learning by the teacher. In this work, we formalize the teaching process with limited feedback and propose an algorithm that solves this teaching problem. The algorithm uses a modified version of the active value-at-risk method to select the starting states, a modified maximum causal entropy algorithm to infer the policy, and the difficulty score ratio method to choose the teaching demonstrations. We test the algorithm in a synthetic car driving environment and conclude that the proposed algorithm is an effective solution when the learner’s feedback is limited.

Paper Citation

in Harvard Style

Zayanov R., Melo F. and Lopes M. (2024). Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 15-24. DOI: 10.5220/0012296800003636

in Bibtex Style

@conference{icaart24,
author={Rustam Zayanov and Francisco Melo and Manuel Lopes},
title={Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART},
year={2024},
pages={15-24},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012296800003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 1: ICAART
TI - Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback
SN - 978-989-758-680-4
AU - Zayanov R.
AU - Melo F.
AU - Lopes M.
PY - 2024
SP - 15
EP - 24
DO - 10.5220/0012296800003636
PB - SciTePress