Reinforcement Learning (RL) is a paradigm where an agent learns sequential actions in uncertain environments, aiming to maximize cumulative rewards. The effectiveness of RL has been established across diverse domains, showcasing its aptitude for handling complex tasks [1,2]. Unlike other machine learning paradigms, RL distinguishes itself by actively acquiring information, with agents gathering data through firsthand interaction with the environment, termed as ‘online’ RL. However, in many real-world scenarios, such as robotics, autonomous driving, and healthcare, direct experimentation can be infeasible or even hazardous. Moreover, direct interactions can often be costly. Thus, it can be beneficial for agents to learn from existing offline datasets, eliminating the need for direct experimentation. This approach, termed ‘offline’ RL or batch RL, focuses on learning exclusively from previously collected datasets.
Offline RL, while advantageous in utilizing existing datasets, presents unique challenges [3]. A significant concern is balancing the enhancement of generalization capabilities and preventing undesired out-of-distribution (OOD) behaviors, largely due to the phenomenon of ‘distributional shift’. During policy evaluation, using Bellman updates can lead to the querying of values from OOD state-action pairs, potentially resulting in a cascade of extrapolation errors. The problem intensifies with the use of high-capacity function approximators, like neural networks. To tackle these OOD issues, several offline RL algorithms incorporate a form of pessimism, employing both model-free and model-based methods [4-8]. Recent model-based approaches adjust the estimated Markov Decision Process (MDP) model from offline datasets to encourage conservative behavior [6,7,9]. They penalize the policy or rewards in situations where the model’s estimated accuracy is low. Various methods have sought to achieve this balance by either estimating model uncertainty or regularizing the value function. However, the questions of which choice of model uncertainty should be used and what conservatism is more suitable in practice still remain open.
In a recent paper by the research group of Prof. Min-hwan Oh published at ICML 2023, a novel method for model-based offline RL, named ‘Count-MORL’, is introduced. This method utilizes the frequency of state-action pairs from offline datasets to quantify model estimation errors, penalizing the reward function based on these estimates. A unique aspect of Count-MORL is its reliance on hash code heuristics to approximate the frequency of state-action pairs, particularly beneficial for continuous state and action spaces. This research underscores the significance of count uncertainty and its potential impact on performance. Numerical evaluations reveal that Count-MORL consistently surpasses existing offline RL methods in terms of performance. The main contributions of this study include the introduction of count-based conservatism in model-based offline deep RL, providing theoretical analyses for the proposed method, and demonstrating superior performance in practical applications.
