Deep learning as optimal control problems: models and numerical methods

Abstract

We consider recent work of [11] and [6], where deep learning neuralnetworks have been interpreted as discretisations of an optimal control problemsubject to an ordinary differential equation constraint. We review the first orderconditions for optimality, and the conditions ensuring optimality after discretiza-tion. This leads to a class of algorithms for solving the discrete optimal controlproblem which guarantee that the corresponding discrete necessary conditions foroptimality are fulfilled. We discuss two different deep learning algorithms and makea preliminary analysis of the ability of the algorithms to generalise.

Deep learning as optimal control problems: models and numerical methods