Evaluating Reinforcement Learning Policies in Observational Healthcare Using Robust Off-Policy Estimation and Diagnostic Methods

Abstract

The increasing integration of machine learning (ML), particularly reinforcement learning (RL), into healthcare has generated significant interest in developing data-driven treatment strategies. However, reliable evaluation of RL policies using retrospective clinical data remains a fundamental challenge, given issues such as data sparsity, high variance in off-policy estimates, and potential biases arising from confounding variables. This study proposes a robust methodological framework for evaluating RL algorithms in observational health settings, with a specific focus on sepsis management using the MIMIC-III database. The framework integrates advanced statistical estimators, including weighted doubly robust (WDR) methods, and incorporates empirical diagnostics such as importance weight distribution analyses and effective sample size calculations. We systematically compare the RL-derived optimal policy against clinician, random, and no-action baselines over 50 randomized train-test splits. Quantitative results demonstrate that while the RL policy achieves higher average cumulative reward estimates, the performance gains are accompanied by substantial variance and limited data support, raising important considerations about the interpretability and generalizability of such models. By explicitly addressing the methodological gaps present in prior works, this research offers a transparent, reproducible, and clinically grounded approach to RL policy evaluation. The findings highlight the necessity of combining algorithmic innovation with rigorous evaluation practices and domain expertise to ensure safe and effective translation of RL systems into real-world clinical workflows. This study contributes both methodological advancements and practical recommendations that can inform future development and validation of machine learning applications in healthcare