Authors:
Jonas Nüßlein
;
Maximilian Zorn
;
Philipp Altmann
and
Claudia Linnhoff-Popien
Affiliation:
Institute of Computer Science, LMU Munich, Germany
Keyword(s):
Reinforcement Learning, Imitation Learning, Ensemble, Robustness.
Abstract:
In sequential decision-making environments, the primary approaches for training agents are Reinforcement Learning (RL) and Imitation Learning (IL). Unlike RL, which relies on modeling a reward function, IL leverages expert demonstrations, where an expert policy πe (e.g., a human) provides the desired behavior. Formally, a dataset D of state-action pairs is provided: D = (s,a = πe(s)). A common technique within IL is Behavior Cloning (BC), where a policy π(s) = a is learned through supervised learning on D. Further improvements can be achieved by using an ensemble of N individually trained BC policies, denoted as E = {πi(s)}1≤i≤N. The ensemble’s action a for a given state s is the aggregated output of the N actions: a = 1 N ∑i πi(s). This paper addresses the issue of increasing action differences—the observation that discrepancies between the N predicted actions grow in states that are underrepresented in the training data. Large action differences can result in suboptimal aggregated
actions. To address this, we propose a method that fosters greater alignment among the policies while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. We evaluate our approach across eight diverse environments, demonstrating a notable decrease in action differences and significant improvements in overall performance, as measured by mean episode returns.
(More)