Deep Reinforcement Learning Applied to Managed Pressure Drilling

Abstract

During drilling operations, maintaining a desired downhole pressure between pressure margins is crucial to avoid damage to the formation and the well. The process is highly nonlinear, changing with depth, and every section in every well is different. Standard solutions with PID controllers are widely accepted for this purpose, although methods such as deep reinforcement learning (DRL) could be investigated as an alternative approach.

A smooth update deep Q learning algorithm is used to train an agent embedded in a managed pressure drilling system. The aim is to control downhole pressure during pipe connections by use of a topside choke valve with nonlinear characteristics. The agent is trained on previously gathered data, from situations featuring step changes in pressure setpoint and changing mud flows, all at various well depths. After training, the agent is tasked with controlling BHP during connection, herein demonstrated by use of a numerically simulated low-order hydraulics model.

Through episodic training, it becomes clear that the agent purely through interaction with the environment, and without any prerequisite knowledge of system dynamics and reward design, converges to an optimal control policy. The trained agent is then tested on pipe connections with well depths in the lower and upper bounds of the training data. The pipe connection scenario presents changes in operating conditions in terms of changing mud flows with changing conditions like increased frictional pressure losses due to increased depth. Still, the results presented show the agent's ability to track a pressure setpoint at various depths in the changing conditions present during connection, while seamlessly incorporating controller constraints.

There are several advantages associated with this approach, among them eliminating the need for development of a complex dynamic model for the process. Also, the approach is applicable to both linear and nonlinear systems, deterministic and stochastic systems, and lower- and higher-level decision-making. These methods could possibly also be applied to other key challenges in drilling such as ROP optimization or autonomous directional drilling.