Theoretical Properties of Learning-based Model Predictive Control

Abstract

Recently, the core idea of using Model Predictive Control (MPC) as a function approximator for the Reinforcement Learning (RL) methods has been proposed and justified. More specifically, it has been shown that a parameterized MPC scheme with a possibly inaccurate model can capture the optimal value functions and policy of a given Markov Decision Process (MDP). The thesis investigates more on this idea and provides theorems supporting and developing this idea and answering some fundamental questions in the intersection of MDP, MPC, Moving Horizon Estimation (MHE), and RL based on the publications during the Ph.D. We implement MPC-based RL in engineering applications such as Autonomous Surface Vehicle (ASV), including path planning, obstacle avoidance, and docking, and some investigations in the smart grid context, including learning the optimal bang-bang policy and multi-agent batteries with power peak constraint. In the intersection of MDP and MPC, we provide a theory on the equivalence of optimality criteria for MPC and MDP. We show that an (undiscounted) MPC scheme can capture the optimal value and optimal policy of a (possibly discounted) MDP, even if an inaccurate model is used in the MPC scheme. This equivalence can be established using a proper selection of the stage cost and the terminal cost of an MPC scheme. This observation leads us to parameterize an MPC scheme fully, including the cost function. In practice, Reinforcement Learning algorithms can then be used to tune the parameterized MPC scheme. Using the cost modification idea, we also eliminate the bias of the optimal steady state in the discounted setting. In the context of MDP and RL, we provide the Quasi-Newton technique with a novel approximated hessian of the performance function that yields a superlinear convergence in the learning using the policy gradient method. In addition, we characterize the stability of MDPs with discounted cost using Economic Model Predictive Control (EMPC) dissipativity theory in the measure space. In the context of EMPC, we propose the use of Q-learning to capture a valid storage function that satisfies the dissipation inequality and verify the dissipativity for both discounted and undiscounted settings. Robust Model Predictive Control (RMPC) is used for different purposes and forms. We address the bias issue in the MPC-based policy gradient method when a linear compatible advantage function approximator is used in the actor-critic. When hard constraints restrict the policy, the exploration may not be Centred or Isotropic (non-CI). As a result, the policy gradient estimation can be biased. We solve this issue using the RMPC approach accounting for the exploration based on the first-order Taylor approximation of the constraint-tightening. Moreover, we investigate using RL methods to adjust RMPC with ellipsoidal uncertainty set for stochastic nonlinear systems. Scenario-tree-based RMPC was implemented to handle potential failures of the ship thrusters and Q-learning was used to improve the closed-loop performance. Moreover, we provide a generic convex function approximator in the stage cost of the MPC scheme and also address the safe RL problem using the Distributionally Robust Model Predictive Control (DRMPC) scheme and chance constraints.