Safe Reinforcement Learning for Autonomy in Resource-Constrained Environments

Abstract

Artificial Intelligence (AI) has rapidly permeated various sectors of society in recent years. From consumer electronics, education, finance, medicine, autonomous driving, and chatbots, AI is transforming the way we live, work, and interact with the world around us. The success of AI follows the recent increase in the availability and capacity of hardware accelerators capable of effectively processing artificial neural networks (ANNs). In the context of control, Deep Reinforcement Learning (DRL) methods leverage ANNs to enable intelligent control where classical methods fail and have attained superhuman performance in several domains as a result. These successes, however, mask several practical challenges. DRL is a computationally expensive approach that is fundamentally based on trial and error and requires a significant amount of attempts to train autonomous agents. This inhibits applying DRL directly in real-world applications, both from a time perspective and, more importantly, a safety perspective. This limits most current approaches to training in simulation. However, given a fully trained DRL agent in simulation, the black-box nature of the ANN parameterization prevents the safety guarantees necessary to apply the agent in safety-critical applications. On the edge, assuming that the safety concerns can be addressed, the computational capacity of hardware restricts the model complexity of the onboard control system, particularly on small platforms like quadrotor drones. This thesis presents topics related to increasing the safety and efficiency of DRL in such resource-constrained systems. It is split into an introduction, three main parts, and a conclusion. The introduction provides context for the publications, describes the considered applications, presents the research questions, and outlines the structure of this thesis. The main parts incrementally introduce new topics and detail the context and preliminaries leading to the presented published works. The first part introduces DRL as a model-free control framework. It establishes Proximal Policy Optimization (PPO) as the best-performing and most versatile DRL algorithm for continuous control through a comparative analysis against other leading algorithms. The second part introduces safe control and presents two articles, each proposing a different approach for improving safety in DRL-based control. Through reward engineering, DRL agents demonstrate compliance with the International Regulations for Preventing Collisions at Sea (COLREG). Safety guarantees are established via a Predictive Safety Filter (PSF), combining model-based and model-free control. The third part moves into DRL for resource-constrained environments and is two-fold in its presentation. First, transfer learning through the latent representation of high-dimensional sensor data is proposed to reduce the computational complexity of DRL. Second, digital twins show great promise toward the training and validation of DRL agents prior to the deployment in their real-world counterparts, inherently minimizing the sim-to-real gap in cluttered indoor environments. Further, surrogate modeling and efficient representations enable the online utilization of digital twins on edge hardware. This enables safe and dynamic path planning and DRL-based control using otherwise immeasurable information in an urban environment. Ultimately, this thesis explores the challenges and proposes advancements in enabling autonomy in resource-constrained environments using DRL. Through this series of research contributions, the work highlights the importance of balancing safety and efficiency to develop autonomous systems capable of operating in complex and dynamic scenarios. Each paper contributes to a layered understanding of how these challenges can be addressed within the DRL framework, culminating in a promising outlook for its position in future real-world autonomy.